May 18 18:33:28.503057 np0000174525 systemd[1]: Started devstack@neutron-ovn-maintenance-worker.service - Devstack devstack@neutron-ovn-maintenance-worker.service. May 18 18:33:28.872676 np0000174525 neutron-ovn-maintenance-worker[105284]: /opt/stack/data/venv/lib/python3.12/site-packages/oslo_concurrency/lockutils.py:41: EventletDeprecationWarning: May 18 18:33:28.872676 np0000174525 neutron-ovn-maintenance-worker[105284]: Eventlet is deprecated. It is currently being maintained in bugfix mode, and May 18 18:33:28.872676 np0000174525 neutron-ovn-maintenance-worker[105284]: we strongly recommend against using it for new projects. May 18 18:33:28.872676 np0000174525 neutron-ovn-maintenance-worker[105284]: If you are already using Eventlet, we recommend migrating to a different May 18 18:33:28.872676 np0000174525 neutron-ovn-maintenance-worker[105284]: framework. For more detail see May 18 18:33:28.872676 np0000174525 neutron-ovn-maintenance-worker[105284]: https://eventlet.readthedocs.io/en/latest/asyncio/migration.html May 18 18:33:28.872676 np0000174525 neutron-ovn-maintenance-worker[105284]: import eventlet May 18 18:33:30.286218 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.common.config [-] Logging enabled! May 18 18:33:30.286819 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-ovn-maintenance-worker version 28.1.0.dev227 May 18 18:33:30.286819 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-ovn-maintenance-worker --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini {{(pid=105284) setup_logging /opt/stack/neutron/neutron/common/config.py:124}} May 18 18:33:30.286819 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.server.ovn_maintenance [-] OVN maintenance process starting... May 18 18:33:30.287133 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG oslo_concurrency.lockutils [-] Acquiring lock "plugin-directory" by "neutron_lib.plugins.directory._create_plugin_directory" {{(pid=105284) inner /opt/stack/data/venv/lib/python3.12/site-packages/oslo_concurrency/lockutils.py:506}} May 18 18:33:30.287133 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG oslo_concurrency.lockutils [-] Lock "plugin-directory" acquired by "neutron_lib.plugins.directory._create_plugin_directory" :: waited 0.000s {{(pid=105284) inner /opt/stack/data/venv/lib/python3.12/site-packages/oslo_concurrency/lockutils.py:519}} May 18 18:33:30.287133 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG oslo_concurrency.lockutils [-] Lock "plugin-directory" "released" by "neutron_lib.plugins.directory._create_plugin_directory" :: held 0.000s {{(pid=105284) inner /opt/stack/data/venv/lib/python3.12/site-packages/oslo_concurrency/lockutils.py:538}} May 18 18:33:30.287133 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG oslo_concurrency.lockutils [-] Acquiring lock "manager" by "neutron.manager.NeutronManager._create_instance" {{(pid=105284) inner /opt/stack/data/venv/lib/python3.12/site-packages/oslo_concurrency/lockutils.py:506}} May 18 18:33:30.287404 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG oslo_concurrency.lockutils [-] Lock "manager" acquired by "neutron.manager.NeutronManager._create_instance" :: waited 0.000s {{(pid=105284) inner /opt/stack/data/venv/lib/python3.12/site-packages/oslo_concurrency/lockutils.py:519}} May 18 18:33:30.287404 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.manager [-] Loading core plugin: ml2 May 18 18:33:30.613199 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_create 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:30.613466 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_update 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:30.613466 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_delete 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:30.614156 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_create 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:30.614156 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_update 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:30.614341 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_delete 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:30.650943 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_create 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:30.650943 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_update 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:30.651103 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_delete 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:30.858445 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_create 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:30.858528 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_update 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:30.858603 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_delete 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:30.899508 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_create 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:30.899658 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_update 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:30.899658 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_delete 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:30.901652 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_create 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:30.901652 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_update 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:30.901805 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_delete 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:30.977305 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: agent after_create 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:30.977305 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: agent after_update 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:30.977305 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: segment precommit_create 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:30.977305 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: network precommit_delete 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:30.977305 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: segment before_delete 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:31.030910 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > network before_delete 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:31.031035 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: agent before_delete 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:31.031079 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > network before_create 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:31.031106 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port before_create 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:31.031171 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port before_update 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:31.031244 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > segment precommit_create 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:31.031299 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > segment precommit_delete 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:31.031366 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > segment after_create 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:31.031423 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > segment after_delete 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:31.031485 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > network after_delete 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:31.031542 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > network precommit_delete 0, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:31.031602 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port provisioning_complete 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:31.031662 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_create 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:31.031719 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy after_delete 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:31.031780 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > agent after_update 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:31.031840 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > subnet after_delete 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:31.031905 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > subnet precommit_delete 0, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:31.031978 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_update 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:31.032036 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_delete 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:31.032103 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > segment_host_mapping after_create 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:31.032173 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > agent after_delete 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:31.032269 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_create 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:31.032329 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_update 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:31.033338 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_delete 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:31.033338 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.quota.resource_registry [-] Tracking information for resource: network configured {{(pid=105284) set_tracked_resource /opt/stack/neutron/neutron/quota/resource_registry.py:203}} May 18 18:33:31.033338 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.quota.resource_registry [-] Tracking information for resource: port configured {{(pid=105284) set_tracked_resource /opt/stack/neutron/neutron/quota/resource_registry.py:203}} May 18 18:33:31.033338 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.quota.resource_registry [-] Tracking information for resource: subnet configured {{(pid=105284) set_tracked_resource /opt/stack/neutron/neutron/quota/resource_registry.py:203}} May 18 18:33:31.033338 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.quota.resource_registry [-] Tracking information for resource: subnetpool configured {{(pid=105284) set_tracked_resource /opt/stack/neutron/neutron/quota/resource_registry.py:203}} May 18 18:33:31.033338 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.quota.resource_registry [-] Tracking information for resource: security_group configured {{(pid=105284) set_tracked_resource /opt/stack/neutron/neutron/quota/resource_registry.py:203}} May 18 18:33:31.033338 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.quota.resource_registry [-] Tracking information for resource: security_group_rule configured {{(pid=105284) set_tracked_resource /opt/stack/neutron/neutron/quota/resource_registry.py:203}} May 18 18:33:31.033523 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.plugins.ml2.managers [-] Configured type driver names: ['local', 'flat', 'vlan', 'geneve'] May 18 18:33:31.034059 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.plugins.ml2.drivers.type_flat [-] Allowable flat physical_network names: ['public'] May 18 18:33:31.034664 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.plugins.ml2.drivers.type_local [-] ML2 LocalTypeDriver initialization complete May 18 18:33:31.035238 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.plugins.ml2.drivers.type_vlan [-] Network VLAN ranges: OrderedDict({'public': [(1, 4094)]}) May 18 18:33:31.035302 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.plugins.ml2.managers [-] Loaded type driver names: ['flat', 'geneve', 'local', 'vlan'] May 18 18:33:31.037337 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.plugins.ml2.managers [-] Registered types: dict_keys(['flat', 'geneve', 'local', 'vlan']) May 18 18:33:31.037337 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.plugins.ml2.managers [-] Project network_types: ['geneve'] May 18 18:33:31.037337 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.plugins.ml2.managers [-] Configured extension driver names: ['port_security', 'qos'] May 18 18:33:31.037337 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.plugins.ml2.managers [-] Loaded extension driver names: ['port_security', 'qos'] May 18 18:33:31.037337 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.plugins.ml2.managers [-] Registered extension drivers: ['port_security', 'qos'] May 18 18:33:31.037337 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.plugins.ml2.managers [-] Configured mechanism driver names: ['ovn'] May 18 18:33:31.073911 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.plugins.ml2.managers [-] Loaded mechanism driver names: ['ovn'] May 18 18:33:31.074035 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.plugins.ml2.managers [-] Registered mechanism drivers: ['ovn'] May 18 18:33:31.074035 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.plugins.ml2.managers [-] No mechanism drivers provide segment reachability information for agent scheduling. May 18 18:33:31.074617 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > floatingip before_response 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:31.074695 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port before_response 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:31.096002 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.plugins.ml2.managers [-] Initializing driver for type 'flat' May 18 18:33:31.096002 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.plugins.ml2.drivers.type_flat [-] ML2 FlatTypeDriver initialization complete May 18 18:33:31.096106 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.plugins.ml2.managers [-] Initializing driver for type 'geneve' May 18 18:33:31.096185 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.plugins.ml2.drivers.type_tunnel [-] geneve ID ranges: [(1, 65536)] May 18 18:33:31.166807 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO dbcounter [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Registered counter for database neutron May 18 18:33:31.199340 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG oslo_db.sqlalchemy.engines [None req-9a75489d-6914-46db-b03b-7df031963107 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=105284) _check_effective_sql_mode /opt/stack/data/venv/lib/python3.12/site-packages/oslo_db/sqlalchemy/engines.py:325}} May 18 18:33:31.801362 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG dbcounter [-] [105284] Writer thread running {{(pid=105284) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:102}} May 18 18:33:32.048987 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.plugins.ml2.managers [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Initializing driver for type 'local' May 18 18:33:32.048987 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.plugins.ml2.managers [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Initializing driver for type 'vlan' May 18 18:33:32.175461 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.plugins.ml2.drivers.type_vlan [None req-9a75489d-6914-46db-b03b-7df031963107 None None] VlanTypeDriver initialization complete May 18 18:33:32.175602 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.plugins.ml2.managers [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Initializing extension driver 'port_security' May 18 18:33:32.175602 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.plugins.ml2.extensions.port_security [None req-9a75489d-6914-46db-b03b-7df031963107 None None] PortSecurityExtensionDriver initialization complete May 18 18:33:32.175795 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.plugins.ml2.managers [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Initializing extension driver 'qos' May 18 18:33:32.175795 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.plugins.ml2.extensions.qos [None req-9a75489d-6914-46db-b03b-7df031963107 None None] QosExtensionDriver initialization complete {{(pid=105284) initialize /opt/stack/neutron/neutron/plugins/ml2/extensions/qos.py:31}} May 18 18:33:32.175921 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.plugins.ml2.managers [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Initializing mechanism driver 'ovn' May 18 18:33:32.176018 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Starting OVNMechanismDriver May 18 18:33:32.177916 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Subscribe: > process before_spawn 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.178047 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Subscribe: > process after_init 55550000, True {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.178137 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Subscribe: > segment after_create 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.178250 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Subscribe: > segment after_create 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.178358 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Subscribe: > segment after_delete 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.178358 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Subscribe: > allowed_address_pair before_create 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.178509 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Subscribe: > security_group precommit_create 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.178572 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Subscribe: > security_group after_update 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.178632 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Subscribe: > security_group after_create 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.178719 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Subscribe: > security_group precommit_delete 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.178783 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Subscribe: > security_group after_delete 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.178849 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Subscribe: > security_group_rule precommit_create 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.178956 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Subscribe: > security_group_rule after_create 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.179019 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Subscribe: > security_group_rule before_delete 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.179100 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Subscribe: > address_group after_create 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.179182 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Subscribe: > address_group after_update 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.179433 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Subscribe: > address_group after_delete 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.179433 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Subscribe: > qos_plugin after_init 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.179656 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Subscribe: > trunk_plugin after_init 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.179857 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Subscribe: > logging-plugin after_init 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.180347 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.services.logapi.drivers.ovn.driver [None req-9a75489d-6914-46db-b03b-7df031963107 None None] OVN logging driver registered May 18 18:33:32.181769 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.plugins.ml2.plugin [None req-9a75489d-6914-46db-b03b-7df031963107 None None] neutron.plugins.ml2.plugin.Ml2Plugin method _start_rpc_notifiers called with arguments () {} {{(pid=105284) wrapper /opt/stack/data/venv/lib/python3.12/site-packages/oslo_log/helpers.py:77}} May 18 18:33:32.182015 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Subscribe: > port after_create 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.182086 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Subscribe: > port after_update 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.182180 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Subscribe: > port after_delete 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.182264 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Subscribe: > subnet after_create 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.182370 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Subscribe: > subnet after_update 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.182434 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Subscribe: > subnet after_delete 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.182545 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Subscribe: > network after_create 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.182609 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Subscribe: > network after_update 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.182669 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Subscribe: > network after_delete 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.182771 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Subscribe: > security_group after_create 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.182834 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Subscribe: > security_group after_update 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.182894 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Subscribe: > security_group after_delete 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.182986 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Subscribe: > security_group_rule after_create 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.183065 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Subscribe: > security_group_rule after_update 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.183126 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Subscribe: > security_group_rule after_delete 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.183211 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Subscribe: > address_group after_create 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.183271 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Subscribe: > address_group after_update 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.183385 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Subscribe: > address_group after_delete 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.183464 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.plugins.ml2.ovo_rpc [None req-9a75489d-6914-46db-b03b-7df031963107 None None] ML2 OVO RPC backend initialized. {{(pid=105284) __init__ /opt/stack/neutron/neutron/plugins/ml2/ovo_rpc.py:117}} May 18 18:33:32.184428 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.quota [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loaded quota_driver: . May 18 18:33:32.184504 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.plugins.ml2.plugin [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Modular L2 Plugin initialization complete May 18 18:33:32.184629 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.manager [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading services supported by the core plugin {{(pid=105284) _load_services_from_core_plugin /opt/stack/neutron/neutron/manager.py:170}} May 18 18:33:32.184737 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.plugins.ml2.managers [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Got port-security extension from driver 'port_security' May 18 18:33:32.184983 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.manager [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading service plugins: ['ovn-router', 'auto_allocate', 'tag', 'timestamp', 'network_ip_availability', 'flavors', 'revisions'] {{(pid=105284) _load_service_plugins /opt/stack/neutron/neutron/manager.py:195}} May 18 18:33:32.185050 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.manager [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading Plugin: ovn-router May 18 18:33:32.195024 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Subscribe: > router before_delete 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.195225 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Subscribe: port before_update 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.195363 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Subscribe: port after_update 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.195488 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Subscribe: > port precommit_delete 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.195625 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Subscribe: port before_update 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.195717 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Subscribe: port before_delete 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.195844 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Subscribe: > router precommit_create 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.195958 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Subscribe: > router precommit_create 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.196155 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Subscribe: > router precommit_update 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.196291 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Subscribe: subnet after_update 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.196649 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.quota.resource_registry [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Tracking information for resource: router configured {{(pid=105284) set_tracked_resource /opt/stack/neutron/neutron/quota/resource_registry.py:203}} May 18 18:33:32.196743 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.quota.resource_registry [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Tracking information for resource: floatingip configured {{(pid=105284) set_tracked_resource /opt/stack/neutron/neutron/quota/resource_registry.py:203}} May 18 18:33:32.196802 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.services.ovn_l3.plugin [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Starting OVNL3RouterPlugin May 18 18:33:32.197036 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Subscribe: > port_forwarding_plugin after_init 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.197212 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Subscribe: > router before_create 55550100, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.197303 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Subscribe: > router precommit_delete 55550100, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.197489 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Subscribe: > router precommit_create 55550100, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.197489 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Subscribe: > router precommit_update 55550100, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.200195 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.services.provider_configuration [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Service providers = [] {{(pid=105284) parse_service_provider_opt /opt/stack/neutron/neutron/services/provider_configuration.py:170}} May 18 18:33:32.200271 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.db.servicetype_db [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Adding provider configuration for service L3_ROUTER_NAT {{(pid=105284) add_provider_configuration /opt/stack/neutron/neutron/db/servicetype_db.py:45}} May 18 18:33:32.201826 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Subscribe: > router precommit_create 55549900, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.201895 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Subscribe: > router_interface after_create 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.201983 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Subscribe: > floatingip after_create 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.202068 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Subscribe: > floatingip precommit_create 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.202159 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Subscribe: > floatingip precommit_update 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.202226 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Subscribe: > floatingip after_delete 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.202299 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Subscribe: > floatingip after_status_update 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.202411 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Subscribe: > floatingip after_update 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.202502 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Subscribe: > port_forwarding after_create 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.202596 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Subscribe: > router_interface after_delete 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.202688 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Subscribe: > router after_create 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.202764 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Subscribe: > router precommit_create 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.202836 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Subscribe: > router after_delete 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.202925 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Subscribe: > router after_update 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.203022 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.services.service_base [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loaded 'neutron.services.ovn_l3.service_providers.ovn.OvnDriver' provider for service L3_ROUTER_NAT {{(pid=105284) load_drivers /opt/stack/neutron/neutron/services/service_base.py:49}} May 18 18:33:32.203155 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Subscribe: > process after_init 55550001, True {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.203304 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.manager [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Successfully loaded L3_ROUTER_NAT plugin. Description: L3 Router Service Plugin for basic L3 forwarding using OVN {{(pid=105284) _create_and_add_service_plugin /opt/stack/neutron/neutron/manager.py:239}} May 18 18:33:32.203374 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.manager [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading Plugin: auto_allocate May 18 18:33:32.204358 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Subscribe: network precommit_update 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.204494 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron_lib.callbacks.manager [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Subscribe: network precommit_create 55550000, False {{(pid=105284) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.205246 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.manager [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Successfully loaded auto-allocated-topology plugin. Description: Auto Allocated Topology - aka get me a network. {{(pid=105284) _create_and_add_service_plugin /opt/stack/neutron/neutron/manager.py:239}} May 18 18:33:32.205246 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.manager [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading Plugin: tag May 18 18:33:32.207805 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.manager [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Successfully loaded TAG plugin. Description: Tag support {{(pid=105284) _create_and_add_service_plugin /opt/stack/neutron/neutron/manager.py:239}} May 18 18:33:32.207917 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.manager [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading Plugin: timestamp May 18 18:33:32.209094 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.manager [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Successfully loaded timestamp plugin. Description: Adds timestamps to Neutron resources with standard attributes {{(pid=105284) _create_and_add_service_plugin /opt/stack/neutron/neutron/manager.py:239}} May 18 18:33:32.209200 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.manager [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading Plugin: network_ip_availability May 18 18:33:32.210642 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.manager [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Successfully loaded network-ip-availability plugin. Description: Provides IP availability data for each network and subnet. {{(pid=105284) _create_and_add_service_plugin /opt/stack/neutron/neutron/manager.py:239}} May 18 18:33:32.210744 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.manager [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading Plugin: flavors May 18 18:33:32.211574 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.manager [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Successfully loaded FLAVORS plugin. Description: Neutron Flavors and Service Profiles manager plugin {{(pid=105284) _create_and_add_service_plugin /opt/stack/neutron/neutron/manager.py:239}} May 18 18:33:32.211665 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.manager [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading Plugin: revisions May 18 18:33:32.212676 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.manager [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Successfully loaded revision_plugin plugin. Description: Adds revision numbers to resources. {{(pid=105284) _create_and_add_service_plugin /opt/stack/neutron/neutron/manager.py:239}} May 18 18:33:32.212778 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG oslo_concurrency.lockutils [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Lock "manager" "released" by "neutron.manager.NeutronManager._create_instance" :: held 1.926s {{(pid=105284) inner /opt/stack/data/venv/lib/python3.12/site-packages/oslo_concurrency/lockutils.py:538}} May 18 18:33:32.213057 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] get_extension_paths = OrderedDict({'/opt/stack/neutron/neutron/extensions': 1}) {{(pid=105284) get_extensions_path /opt/stack/neutron/neutron/api/extensions.py:640}} May 18 18:33:32.213212 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Initializing extension manager. May 18 18:33:32.213529 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: __init__.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.213529 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: __pycache__ {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.213738 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: address_group.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.213981 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Address group" alias="address-group" description="Support address group" updated="2020-07-15T10:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.214305 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loaded extension: address-group May 18 18:33:32.214418 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: address_scope.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.214731 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Address scope" alias="address-scope" description="Address scopes extension." updated="2015-07-26T10:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.214875 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loaded extension: address-scope May 18 18:33:32.214958 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: admin_state_down_before_update.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.215135 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Enforce Router's Admin State Down Before Update Extension" alias="router-admin-state-down-before-update" description="Ensure that the admin state of a router is DOWN (admin_state_up=False) before updating the distributed attribute" updated="2019-07-02T15:56:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.215279 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Extension router-admin-state-down-before-update not supported by any of loaded plugins May 18 18:33:32.215391 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: agent.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.215612 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="agent" alias="agent" description="The agent management extension." updated="2013-02-03T10:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.215746 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loaded extension: agent May 18 18:33:32.215805 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: agent_resources_synced.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.215999 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Agent's Resource View Synced to Placement" alias="agent-resources-synced" description="Stores success/failure of last sync to Placement" updated="2018-12-19T00:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.216159 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Extension agent-resources-synced not supported by any of loaded plugins May 18 18:33:32.216220 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: agent_sort_key.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.216436 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Agents Sort Key" alias="agent_sort_key" description="Enabling the keyword sort_key for sorting in agents." updated="2013-02-03T10:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.216571 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Extension agent_sort_key not supported by any of loaded plugins May 18 18:33:32.216630 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: allowedaddresspairs.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.217291 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Allowed Address Pairs" alias="allowed-address-pairs" description="Provides allowed address pairs" updated="2013-07-23T10:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.217450 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loaded extension: allowed-address-pairs May 18 18:33:32.217509 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: auto_allocated_topology.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.217722 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Auto Allocated Topology Services" alias="auto-allocated-topology" description="Auto Allocated Topology Services." updated="2016-01-01T00:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.217860 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loaded extension: auto-allocated-topology May 18 18:33:32.217919 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: availability_zone.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.218177 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Availability Zone" alias="availability_zone" description="The availability zone extension." updated="2015-01-01T10:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.218311 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loaded extension: availability_zone May 18 18:33:32.218402 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: availability_zone_filter.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.218591 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Availability Zone Filter Extension" alias="availability_zone_filter" description="Add filter parameters to AvailabilityZone resource" updated="2018-06-22T10:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.218721 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Extension availability_zone_filter not supported by any of loaded plugins May 18 18:33:32.218777 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: data_plane_status.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.218990 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Port data plane status extension" alias="data-plane-status" description="Expose status of underlying data plane" updated="2017-01-24T10:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.219132 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Extension data-plane-status not supported by any of loaded plugins May 18 18:33:32.219191 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: default_subnetpools.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.219405 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Default Subnetpools" alias="default-subnetpools" description="Provides ability to mark and use a subnetpool as the default." updated="2016-02-18T18:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.219542 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loaded extension: default-subnetpools May 18 18:33:32.219610 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: dhcpagentscheduler.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.219953 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="DHCP Agent Scheduler" alias="dhcp_agent_scheduler" description="Schedule networks among dhcp agents" updated="2013-02-07T10:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.220142 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loaded extension: dhcp_agent_scheduler May 18 18:33:32.220205 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: dns.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.220419 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="DNS Integration" alias="dns-integration" description="Provides integration with DNS." updated="2015-08-15T18:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.220542 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loaded extension: dns-integration May 18 18:33:32.220600 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: dns_domain_ports.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.220798 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="dns_domain for ports" alias="dns-domain-ports" description="Allows the DNS domain to be specified for a network port." updated="2017-04-24T10:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.220921 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loaded extension: dns-domain-ports May 18 18:33:32.220978 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: dns_integration_domain_keywords.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.221184 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="DNS domain names with keywords allowed" alias="dns-integration-domain-keywords" description="Allows to use keywords like , , and as DNS domain name" updated="2020-06-15T18:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.221307 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loaded extension: dns-integration-domain-keywords May 18 18:33:32.221378 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: dvr.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.221646 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Distributed Virtual Router" alias="dvr" description="Enables configuration of Distributed Virtual Routers." updated="2014-06-1T10:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.221774 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Extension dvr not supported by any of loaded plugins May 18 18:33:32.221849 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: ecmp_routes.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.222059 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Equal Cost Multipath Routing" alias="ecmp_routes" description="Allows traffic to reach thesame destination via multiple different links " updated="2020-11-26T18:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.222196 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Extension ecmp_routes not supported by any of loaded plugins May 18 18:33:32.222253 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: empty_string_filtering.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.222462 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Empty String Filtering Extension" alias="empty-string-filtering" description="Allow filtering by attributes with empty string value" updated="2018-05-01T10:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.222597 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loaded extension: empty-string-filtering May 18 18:33:32.222653 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: expose_l3_conntrack_helper.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.222842 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Expose CT target rules for conntrack helper" alias="expose-l3-conntrack-helper" description="Expose allow adding CT target rules for conntrack helper" updated="2019-04-04T10:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.222971 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Extension expose-l3-conntrack-helper not supported by any of loaded plugins May 18 18:33:32.223029 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: expose_port_forwarding_in_fip.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.223216 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Expose Floating IP port forwardings" alias="expose-port-forwarding-in-fip" description="Expose port forwarding fields in floating IP response" updated="2018-06-28T16:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.223360 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Extension expose-port-forwarding-in-fip not supported by any of loaded plugins May 18 18:33:32.223417 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: external_net.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.223606 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Neutron external network" alias="external-net" description="Adds external network attribute to network resource." updated="2013-01-14T10:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.223739 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loaded extension: external-net May 18 18:33:32.223844 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: extra_dhcp_opt.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.224099 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Neutron Extra DHCP options" alias="extra_dhcp_opt" description="Extra options configuration for DHCP. For example PXE boot options to DHCP clients can be specified (e.g. tftp-server, server-ip-address, bootfile-name)" updated="2013-03-17T12:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.224269 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loaded extension: extra_dhcp_opt May 18 18:33:32.224377 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: extraroute.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.224603 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Neutron Extra Route" alias="extraroute" description="Extra routes configuration for L3 router" updated="2013-02-01T10:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.224770 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loaded extension: extraroute May 18 18:33:32.224863 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: extraroute_atomic.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.225111 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Atomically add/remove extra routes" alias="extraroute-atomic" description="Edit extra routes of a router on server side by atomically adding/removing extra routes" updated="2019-07-10T00:00:00+00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.225272 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loaded extension: extraroute-atomic May 18 18:33:32.225356 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: filter_validation.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.225600 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Filter parameters validation" alias="filter-validation" description="Provides validation on filter parameters." updated="2018-07-04T10:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.225765 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loaded extension: filter-validation May 18 18:33:32.225863 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: fip_pf_description.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.226091 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Floating IP Port Forwarding new attribute description" alias="floating-ip-port-forwarding-description" description="Add a description field to Port Forwarding rules" updated="2019-11-01T10:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.226248 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Extension floating-ip-port-forwarding-description not supported by any of loaded plugins May 18 18:33:32.226375 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: fip_pf_detail.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.226569 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Floating IP port forwarding detail" alias="floating-ip-port-forwarding-detail" description="Allow list floating ip return more port forwarding data, include'id' and 'internal_port_id'" updated="2022-05-06T10:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.226705 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loaded extension: floating-ip-port-forwarding-detail May 18 18:33:32.226762 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: fip_pf_port_range.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.226952 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Floating IP Port Forwarding support port ranges" alias="floating-ip-port-forwarding-port-ranges" description="Add support to port ranges" updated="2020-07-01T10:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.227314 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Extension floating-ip-port-forwarding-port-ranges not supported by any of loaded plugins May 18 18:33:32.227439 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: fip_port_details.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.227673 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Floating IP Port Details Extension" alias="fip-port-details" description="Add port_details attribute to Floating IP resource" updated="2018-04-09T10:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.227958 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loaded extension: fip-port-details May 18 18:33:32.227958 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: flavors.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.228194 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Neutron Service Flavors" alias="flavors" description="Flavor specification for Neutron advanced services." updated="2015-09-17T10:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.228387 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loaded extension: flavors May 18 18:33:32.228453 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: floating_ip_port_forwarding.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.228754 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Floating IP Port Forwarding" alias="floating-ip-port-forwarding" description="Allow port forwarding from floating IP port to an internal port" updated="2018-01-19T10:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.228888 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Extension floating-ip-port-forwarding not supported by any of loaded plugins May 18 18:33:32.228946 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: floatingip_pools.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.229177 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Floating IP Pools Extension" alias="floatingip-pools" description="Provides a floating IP pools API." updated="2018-03-21T10:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.229315 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loaded extension: floatingip-pools May 18 18:33:32.229405 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: ip_allocation.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.229596 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="IP Allocation" alias="ip_allocation" description="IP allocation extension." updated="2016-06-10T23:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.229730 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Extension ip_allocation not supported by any of loaded plugins May 18 18:33:32.229789 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: ip_substring_port_filtering.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.229987 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="IP address substring filtering" alias="ip-substring-filtering" description="Provides IP address substring filtering when listing ports" updated="2017-11-28T09:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.230113 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Extension ip-substring-filtering not supported by any of loaded plugins May 18 18:33:32.230172 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: l2_adjacency.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.230370 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="L2 Adjacency" alias="l2_adjacency" description="Display L2 Adjacency for Neutron Networks." updated="2016-04-12T16:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.230500 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Extension l2_adjacency not supported by any of loaded plugins May 18 18:33:32.230570 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: l3.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.230972 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Neutron L3 Router" alias="router" description="Router abstraction for basic L3 forwarding between L2 Neutron networks and access to external networks via a NAT gateway." updated="2012-07-20T10:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.231108 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loaded extension: router May 18 18:33:32.231176 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: l3_conntrack_helper.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.231761 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="L3 Conntrack helper" alias="l3-conntrack-helper" description="Allow adding CT target rules for conntrack helper" updated="2019-04-04T10:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.231914 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Extension l3-conntrack-helper not supported by any of loaded plugins May 18 18:33:32.231995 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: l3_enable_default_route_bfd.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.232273 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Enable Default Route BFD" alias="enable-default-route-bfd" description="Enables configurable BFD behavior for default routes" updated="2023-02-27T00:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.232441 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loaded extension: enable-default-route-bfd May 18 18:33:32.232495 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: l3_enable_default_route_ecmp.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.232734 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Enable Default Route ECMP" alias="enable-default-route-ecmp" description="Enables configurable ECMP behavior for default routes" updated="2023-02-27T00:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.232866 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loaded extension: enable-default-route-ecmp May 18 18:33:32.232944 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: l3_ext_gw_mode.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.233162 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Neutron L3 Configurable external gateway mode" alias="ext-gw-mode" description="Extension of the router abstraction for specifying whether SNAT should occur on the external gateway" updated="2013-03-28T10:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.233332 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loaded extension: ext-gw-mode May 18 18:33:32.233419 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: l3_ext_ha_mode.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.233650 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="HA Router extension" alias="l3-ha" description="Adds HA capability to routers." updated="2014-04-26T00:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.233788 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loaded extension: l3-ha May 18 18:33:32.233863 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: l3_ext_ndp_proxy.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.234040 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Router support to enable NDP proxy" alias="l3-ext-ndp-proxy" description="Router extension to enable ndp proxy" updated="2020-08-25T00:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.234174 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Extension l3-ext-ndp-proxy not supported by any of loaded plugins May 18 18:33:32.234238 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: l3_extra_gws.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.234434 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Neutron L3 External Gateway Multihoming" alias="external-gateway-multihoming" description="Allow multiple external gateway ports per router" updated="2023-01-18T00:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.234580 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loaded extension: external-gateway-multihoming May 18 18:33:32.234668 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: l3_flavors.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.234861 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Router Flavor Extension" alias="l3-flavors" description="Flavor support for routers." updated="2016-05-17T00:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.234992 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loaded extension: l3-flavors May 18 18:33:32.235061 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: l3_ndp_proxy.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.235306 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="name" alias="l3-ndp-proxy" description="description" updated="2021-06-29T10:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.235446 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Extension l3-ndp-proxy not supported by any of loaded plugins May 18 18:33:32.235517 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: l3_port_ip_change_not_allowed.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.235714 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Prevent L3 router ports IP address change extension" alias="l3-port-ip-change-not-allowed" description="Prevent change of IP address for some L3 router ports" updated="2018-10-09T10:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.235894 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Extension l3-port-ip-change-not-allowed not supported by any of loaded plugins May 18 18:33:32.235974 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: l3agentscheduler.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.236532 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="L3 Agent Scheduler" alias="l3_agent_scheduler" description="Schedule routers among l3 agents" updated="2013-02-07T10:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.236682 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Extension l3_agent_scheduler not supported by any of loaded plugins May 18 18:33:32.236749 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: local_ip.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.237025 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Local IP" alias="local_ip" description="Support Node Local Virtual IP" updated="2021-07-26T10:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.237158 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Extension local_ip not supported by any of loaded plugins May 18 18:33:32.237229 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: logging.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.237481 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Logging API Extension" alias="logging" description="Provides a logging API for resources such as security group and firewall." updated="2017-01-01T10:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.237625 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Extension logging not supported by any of loaded plugins May 18 18:33:32.237693 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: metering.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.238146 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Neutron Metering" alias="metering" description="Neutron Metering extension." updated="2013-06-12T10:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.238328 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Extension metering not supported by any of loaded plugins May 18 18:33:32.238437 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: metering_source_and_destination_fields.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.238696 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Neutron Metering with source and destination filtering" alias="metering_source_and_destination_fields" description="Neutron Metering extension that enables the use of source and destination IP prefixes to create metering label rules." updated="2020-08-21T08:12:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.238847 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Extension metering_source_and_destination_fields not supported by any of loaded plugins May 18 18:33:32.238929 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: multiprovidernet.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.239163 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Multi Provider Network" alias="multi-provider" description="Expose mapping of virtual networks to multiple physical networks" updated="2013-06-27T10:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.239299 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loaded extension: multi-provider May 18 18:33:32.239391 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: netmtu.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.239576 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Network MTU" alias="net-mtu" description="Provides MTU attribute for a network resource." updated="2015-03-25T10:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.239710 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loaded extension: net-mtu May 18 18:33:32.239768 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: netmtu_writable.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.239955 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Network MTU (writable)" alias="net-mtu-writable" description="Provides a writable MTU attribute for a network resource." updated="2017-07-12T00:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.240115 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loaded extension: net-mtu-writable May 18 18:33:32.240176 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: network_availability_zone.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.240404 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Network Availability Zone" alias="network_availability_zone" description="Availability zone support for network." updated="2015-01-01T10:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.240538 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loaded extension: network_availability_zone May 18 18:33:32.240735 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: network_ha.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.240793 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Network HA creation flag" alias="network_ha" description="Network high availability creation flag." updated="2023-04-27T10:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.240915 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Extension network_ha not supported by any of loaded plugins May 18 18:33:32.240972 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: network_ip_availability.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.241202 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Network IP Availability" alias="network-ip-availability" description="Provides IP availability data for each network and subnet." updated="2015-09-24T00:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.241371 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loaded extension: network-ip-availability May 18 18:33:32.241428 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: network_ip_availability_details.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.241645 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Network IP Availability Details Extension" alias="network-ip-availability-details" description="Add ip_availability_details attribute to network-ip-availabilities resource." updated="2025-10-31T00:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.241782 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loaded extension: network-ip-availability-details May 18 18:33:32.241839 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: network_segment_range.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.242147 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Neutron Network Segment Range" alias="network-segment-range" description="Provides support for the network segment range management" updated="2018-11-29T00:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.242277 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Extension network-segment-range not supported by any of loaded plugins May 18 18:33:32.242352 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: pagination.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.242549 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Pagination support" alias="pagination" description="Extension that indicates that pagination is enabled." updated="2016-06-12T00:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.242647 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loaded extension: pagination May 18 18:33:32.242718 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: port_device_profile.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.242909 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Port device profile" alias="port-device-profile" description="Expose the port device profile (Cyborg)" updated="2020-12-17T10:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.243065 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loaded extension: port-device-profile May 18 18:33:32.243141 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: port_hardware_offload_type.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.243334 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Port hardware offload" alias="port-hardware-offload-type" description="Expose port hardware offload extension" updated="2023-05-09T10:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.243456 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Extension port-hardware-offload-type not supported by any of loaded plugins May 18 18:33:32.243513 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: port_hint_ovs_tx_steering.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.243721 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Port hint: Open vSwitch Tx steering" alias="port-hint-ovs-tx-steering" description="Enable Port hint: Open vSwitch Tx steering" updated="2023-01-01T00:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.243853 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Extension port-hint-ovs-tx-steering not supported by any of loaded plugins May 18 18:33:32.243919 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: port_hints.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.244165 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Port hints" alias="port-hints" description="Backend specific port hints to allow backend specific tuning" updated="2023-01-01T00:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.244296 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Extension port-hints not supported by any of loaded plugins May 18 18:33:32.244419 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: port_mac_address_override.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.244660 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Neutron Port MAC address override" alias="port-mac-override" description="Allow overriding the MAC address of a direct-physical Port via the active binding profile" updated="2022-03-18T10:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.244822 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Extension port-mac-override not supported by any of loaded plugins May 18 18:33:32.244902 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: port_mac_address_regenerate.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.245106 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Neutron Port MAC address regenerate" alias="port-mac-address-regenerate" description="Network port MAC address regenerate" updated="2018-05-03T10:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.245272 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loaded extension: port-mac-address-regenerate May 18 18:33:32.245371 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: port_numa_affinity_policy.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.245606 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Port NUMA affinity policy" alias="port-numa-affinity-policy" description="Expose the port NUMA affinity policy" updated="2020-07-08T10:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.245788 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loaded extension: port-numa-affinity-policy May 18 18:33:32.245884 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: port_numa_affinity_policy_socket.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.246114 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Port NUMA affinity policy "socket"" alias="port-numa-affinity-policy-socket" description="Adds "socket" to the supported port NUMA affinity policies" updated="2024-02-09T10:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.246290 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loaded extension: port-numa-affinity-policy-socket May 18 18:33:32.246403 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: port_resource_request.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.246642 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Port Resource Request" alias="port-resource-request" description="Expose resource request to Port" updated="2018-05-08T10:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.246811 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Extension port-resource-request not supported by any of loaded plugins May 18 18:33:32.246885 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: port_resource_request_groups.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.247085 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Port Resource Request Groups" alias="port-resource-request-groups" description="Support requesting multiple groups of resources and traits from the same RP subtree in resource_request" updated="2021-08-02T10:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.247218 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Extension port-resource-request-groups not supported by any of loaded plugins May 18 18:33:32.247275 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: port_trusted_vif.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.247484 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Port trusted vif" alias="port-trusted-vif" description="Expose port 'trusted' attribute in the API" updated="2024-07-10T10:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.247627 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Extension port-trusted-vif not supported by any of loaded plugins May 18 18:33:32.247701 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: portbindings.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.247878 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Port Binding" alias="binding" description="Expose port bindings of a virtual port to external application" updated="2014-02-03T10:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.248009 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loaded extension: binding May 18 18:33:32.248104 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: portbindings_extended.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.248341 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Port Bindings Extended" alias="binding-extended" description="Expose port bindings of a virtual port to external application" updated="2017-07-17T10:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.248471 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loaded extension: binding-extended May 18 18:33:32.248528 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: portsecurity.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.248732 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Port Security" alias="port-security" description="Provides port security" updated="2012-07-23T10:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.248886 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loaded extension: port-security May 18 18:33:32.248960 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: project_id.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.249177 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="project_id field enabled" alias="project-id" description="Extension that indicates that project_id field is enabled." updated="2016-09-09T09:09:09-09:09" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.249256 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loaded extension: project-id May 18 18:33:32.249330 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: providernet.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.249522 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Provider Network" alias="provider" description="Expose mapping of virtual networks to physical networks" updated="2012-09-07T10:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.249670 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loaded extension: provider May 18 18:33:32.249741 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: qinq.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.249990 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="QinQ networks" alias="qinq" description="Expose network 'qinq' attribute in the API. This allows to configure network to allow vlan in vlan configuration using 0x8a88 ethertype (QinQ)." updated="2024-12-02T10:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.250132 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loaded extension: qinq May 18 18:33:32.250205 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: qos.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.251053 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Quality of Service" alias="qos" description="The Quality of Service extension." updated="2015-06-08T10:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.251192 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Extension qos not supported by any of loaded plugins May 18 18:33:32.251255 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: qos_bw_limit_direction.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.251512 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Direction for QoS bandwidth limit rule" alias="qos-bw-limit-direction" description="Allow to configure QoS bandwidth limit rule with specific direction: ingress or egress" updated="2017-04-10T10:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.251659 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Extension qos-bw-limit-direction not supported by any of loaded plugins May 18 18:33:32.251712 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: qos_bw_minimum_ingress.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.251914 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Ingress direction for QoS minimum bandwidth rule" alias="qos-bw-minimum-ingress" description="Allow to configure QoS minumum bandwidth rule with ingress direction." updated="2018-07-09T10:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.252082 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Extension qos-bw-minimum-ingress not supported by any of loaded plugins May 18 18:33:32.252143 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: qos_default.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.252398 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="QoS default policy" alias="qos-default" description="Expose the QoS default policy per project" updated="2017-041-06T10:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.252545 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Extension qos-default not supported by any of loaded plugins May 18 18:33:32.252643 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: qos_fip.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.252836 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Floating IP QoS" alias="qos-fip" description="The floating IP Quality of Service extension" updated="2017-07-20T00:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.252970 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Extension qos-fip not supported by any of loaded plugins May 18 18:33:32.253040 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: qos_fip_network_policy.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.253217 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="QoS floating IP network policy ID" alias="qos-fip-network-policy" description="Adds a the QoS network ID to the floating IP definition" updated="2021-11-15T10:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.253364 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Extension qos-fip-network-policy not supported by any of loaded plugins May 18 18:33:32.253435 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: qos_gateway_ip.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.253616 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Router gateway IP QoS" alias="qos-gateway-ip" description="The Router gateway IP Quality of Service extension" updated="2018-02-24T00:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.253750 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Extension qos-gateway-ip not supported by any of loaded plugins May 18 18:33:32.253817 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: qos_port_network_policy.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.253995 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="QoS port network policy ID" alias="qos-port-network-policy" description="Adds a the QoS network ID to the port definition" updated="2019-11-01T10:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.254158 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Extension qos-port-network-policy not supported by any of loaded plugins May 18 18:33:32.254241 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: qos_pps_minimum_rule.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.254456 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="QoS minimum packet rate rule" alias="qos-pps-minimum" description="Allow to configure QoS minimum packet rate rule." updated="2021-07-14T10:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.254589 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Extension qos-pps-minimum not supported by any of loaded plugins May 18 18:33:32.254657 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: qos_pps_minimum_rule_alias.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.254845 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="QoS minimum packet rate rule alias" alias="qos-pps-minimum-rule-alias" description="API to enable GET, PUT and DELETE operations on QoS minimum packet rate rule without specifying policy ID" updated="2021-10-22T10:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.254976 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Extension qos-pps-minimum-rule-alias not supported by any of loaded plugins May 18 18:33:32.255041 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: qos_pps_rule.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.255237 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="QoS Rule Type Packet per Second Extension" alias="qos-pps" description="Add QoS Rule Type Packet per Second" updated="2021-05-12T10:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.255381 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Extension qos-pps not supported by any of loaded plugins May 18 18:33:32.255450 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: qos_rule_type_details.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.255630 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Details of QoS rule types" alias="qos-rule-type-details" description="Expose details about QoS rule types supported by loaded backend drivers" updated="2017-06-22T10:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.255762 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Extension qos-rule-type-details not supported by any of loaded plugins May 18 18:33:32.255829 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: qos_rule_type_filter.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.256009 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Allow to filter the list of QoS rule types" alias="qos-rule-type-filter" description="Allows to filter the QoS rule type list adding two new flags. "all_rules" prints all implemented QoS rule types."all_supported" prints all supported QoS rule types by the loaded mechanism drivers" updated="2022-02-02T10:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.256201 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Extension qos-rule-type-filter not supported by any of loaded plugins May 18 18:33:32.256272 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: qos_rules_alias.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.256487 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Quality of Service rules alias API" alias="qos-rules-alias" description="API to enable GET, PUT and DELETE operations on QoS policy rules without specifying policy ID" updated="2018-10-07T10:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.256618 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Extension qos-rules-alias not supported by any of loaded plugins May 18 18:33:32.256691 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: quota_check_limit.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.256869 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Quota engine limit check" alias="quota-check-limit" description="Support for checking the resource usage before applying a new quota limit" updated="2021-09-08T16:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.256999 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loaded extension: quota-check-limit May 18 18:33:32.257067 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: quota_check_limit_default.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.257246 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Quota engine limit check by default" alias="quota-check-limit-default" description="By default, the Neutron quota engine checks the resource usage before applying a new quota limit" updated="2024-08-21T16:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.257386 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loaded extension: quota-check-limit-default May 18 18:33:32.257455 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: quotasv2.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.257951 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Quota management support" alias="quotas" description="Expose functions for quotas management per project" updated="2012-07-29T10:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.258026 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loaded extension: quotas May 18 18:33:32.258090 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: quotasv2_detail.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.258593 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Quota details management support" alias="quota_details" description="Expose functions for quotas usage statistics per project" updated="2017-02-10T10:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.258711 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loaded extension: quota_details May 18 18:33:32.258774 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: rbac.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.259261 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="RBAC Policies" alias="rbac-policies" description="Allows creation and modification of policies that control project access to resources." updated="2015-06-17T12:15:12-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.259430 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loaded extension: rbac-policies May 18 18:33:32.259496 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: rbac_address_group.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.259735 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Add address_group type to RBAC" alias="rbac-address-group" description="Add address_group type to network RBAC" updated="2021-01-20T00:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.259878 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Extension rbac-address-group not supported by any of loaded plugins May 18 18:33:32.259947 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: rbac_address_scope.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.260162 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Add address_scope type to RBAC" alias="rbac-address-scope" description="Add address_scope type to RBAC" updated="2020-02-12T00:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.260293 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loaded extension: rbac-address-scope May 18 18:33:32.260380 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: rbac_security_groups.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.260570 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Add security_group type to network RBAC" alias="rbac-security-groups" description="Add security_group type to network RBAC" updated="2019-02-14T00:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.260693 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loaded extension: rbac-security-groups May 18 18:33:32.260762 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: rbac_subnetpool.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.260942 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Add subnetpool type to RBAC" alias="rbac-subnetpool" description="Add subnetpool type to RBAC" updated="2020-02-05T00:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.261075 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Extension rbac-subnetpool not supported by any of loaded plugins May 18 18:33:32.261146 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: revisionifmatch.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.261329 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="If-Match constraints based on revision_number" alias="revision-if-match" description="Extension indicating that If-Match based on revision_number is supported." updated="2016-12-11T00:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.261475 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loaded extension: revision-if-match May 18 18:33:32.261544 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: revisions.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.261749 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Resource revision numbers" alias="standard-attr-revisions" description="This extension will display the revision number of neutron resources." updated="2016-04-11T10:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.261880 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loaded extension: standard-attr-revisions May 18 18:33:32.261971 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: router_availability_zone.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.262216 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Router Availability Zone" alias="router_availability_zone" description="Availability zone support for router." updated="2015-01-01T10:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.262378 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loaded extension: router_availability_zone May 18 18:33:32.262449 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: router_enable_snat.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.262629 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Exposes the router "enable_snat" read-only configuration flag" alias="router-enable-snat" description="Exposes the router "enable_snat" read-only configuration flag" updated="2025-09-18T18:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.262781 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loaded extension: router-enable-snat May 18 18:33:32.262853 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: routerservicetype.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.263030 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Router Service Type" alias="router-service-type" description="Provides router service type" updated="2013-01-29T00:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.263161 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Extension router-service-type not supported by any of loaded plugins May 18 18:33:32.263229 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: security_groups_default_rules.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.263659 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Default rules for security groups" alias="security-groups-default-rules" description="Configure set of security group rules used as default rules for every new security group" updated="2022-12-19T10:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.263797 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loaded extension: security-groups-default-rules May 18 18:33:32.263865 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: security_groups_normalized_cidr.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.264074 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Normalized CIDR field for security group rules" alias="security-groups-normalized-cidr" description="Add new field with normalized remote_ip_prefix cidr in SG rule" updated="2020-07-28T10:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.264214 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loaded extension: security-groups-normalized-cidr May 18 18:33:32.264280 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: security_groups_port_filtering.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.264488 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Port filtering on security groups" alias="port-security-groups-filtering" description="Provides security groups filtering when listing ports" updated="2018-01-09T09:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.264620 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Extension port-security-groups-filtering not supported by any of loaded plugins May 18 18:33:32.264681 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: security_groups_remote_address_group.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.264879 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Remote address group id field for security group rules" alias="security-groups-remote-address-group" description="Add new field of remote address group id in SG rules" updated="2020-08-25T10:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.265001 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loaded extension: security-groups-remote-address-group May 18 18:33:32.265058 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: security_groups_rules_belongs_to_default_sg.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.265245 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Security group rule belongs to the project's default security group" alias="security-groups-rules-belongs-to-default-sg" description="Flag to determine if the security group rule belongs to the project's default security group" updated="2023-05-23T10:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.265389 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loaded extension: security-groups-rules-belongs-to-default-sg May 18 18:33:32.265459 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: security_groups_shared_filtering.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.265632 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Security group filtering on the shared field" alias="security-groups-shared-filtering" description="Support filtering security groups on the shared field" updated="2021-10-05T09:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.265764 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loaded extension: security-groups-shared-filtering May 18 18:33:32.265830 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: securitygroup.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.267444 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="security-group" alias="security-group" description="The security groups extension." updated="2012-10-05T10:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.267591 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loaded extension: security-group May 18 18:33:32.267652 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: segment.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.267918 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Segment" alias="segment" description="Segments extension." updated="2016-02-24T17:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.268108 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Extension segment not supported by any of loaded plugins May 18 18:33:32.268169 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: segments_peer_subnet_host_routes.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.268381 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Segments peer-subnet host routes" alias="segments-peer-subnet-host-routes" description="Add host routes to subnets on a routed network (segments)" updated="2018-06-12T10:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.268508 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Extension segments-peer-subnet-host-routes not supported by any of loaded plugins May 18 18:33:32.268566 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: servicetype.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.268820 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Neutron Service Type Management" alias="service-type" description="API for retrieving service providers for Neutron advanced services" updated="2013-01-20T00:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.268951 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loaded extension: service-type May 18 18:33:32.269007 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: sorting.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.269209 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Sorting support" alias="sorting" description="Extension that indicates that sorting is enabled." updated="2016-06-12T00:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.269275 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loaded extension: sorting May 18 18:33:32.269381 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: standard_attr_segment.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.269580 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Standard Attribute Segment Extension" alias="standard-attr-segment" description="Add standard attributes to Segment resource" updated="2018-04-09T10:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.269714 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Extension standard-attr-segment not supported by any of loaded plugins May 18 18:33:32.269772 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: standardattrdescription.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.269990 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="standard-attr-description" alias="standard-attr-description" description="Extension to add descriptions to standard attributes" updated="2016-02-10T10:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.270128 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loaded extension: standard-attr-description May 18 18:33:32.270195 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: stateful_security_group.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.270395 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Stateful security group" alias="stateful-security-group" description="Indicates if the security group is stateful or not" updated="2019-11-26T09:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.270576 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loaded extension: stateful-security-group May 18 18:33:32.270650 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: stdattrs_common.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.270825 np0000174525 neutron-ovn-maintenance-worker[105284]: WARNING neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Did not find expected name "Stdattrs_common" in /opt/stack/neutron/neutron/extensions/stdattrs_common.py May 18 18:33:32.270930 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: subnet_dns_publish_fixed_ip.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.271160 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Subnet DNS publish fixed IP" alias="subnet-dns-publish-fixed-ip" description="Support choosing to publish DNS records for IPs from a subnet" updated="2019-05-24T10:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.271296 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loaded extension: subnet-dns-publish-fixed-ip May 18 18:33:32.271366 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: subnet_external_network.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.271558 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Subnet belongs to an external network" alias="subnet-external-network" description="Informs if the subnet belongs to an external network" updated="2024-02-05T18:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.271688 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loaded extension: subnet-external-network May 18 18:33:32.271752 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: subnet_onboard.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.271955 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Subnet Onboard" alias="subnet_onboard" description="Provides support for onboarding subnets into subnet pools" updated="2018-12-18T09:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.272123 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Extension subnet_onboard not supported by any of loaded plugins May 18 18:33:32.272225 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: subnet_segmentid_writable.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.272425 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Subnet SegmentID (writable)" alias="subnet-segmentid-writable" description="Provides a writable segment_id attribute for a subnet resource." updated="2018-03-12T00:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.272558 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Extension subnet-segmentid-writable not supported by any of loaded plugins May 18 18:33:32.272615 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: subnet_service_types.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.272801 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Subnet service types" alias="subnet-service-types" description="Provides ability to set the subnet service_types field" updated="2016-03-15T18:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.273114 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loaded extension: subnet-service-types May 18 18:33:32.273198 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: subnetallocation.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.273686 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Subnet Allocation" alias="subnet_allocation" description="Enables allocation of subnets from a subnet pool" updated="2015-03-30T10:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.273844 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loaded extension: subnet_allocation May 18 18:33:32.273902 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: subnetpool_prefix_ops.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.274145 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Subnet Pool Prefix Operations" alias="subnetpool-prefix-ops" description="Provides support for adjusting the prefix list of subnet pools" updated="2019-02-08T10:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.274284 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loaded extension: subnetpool-prefix-ops May 18 18:33:32.274359 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: tag_creation.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.274560 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Tag creation extension" alias="tag-creation" description="Allow to create multiple tags for a resource" updated="2024-23-07T10:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.274728 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loaded extension: tag-creation May 18 18:33:32.274805 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: tag_ports_during_bulk_creation.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.275000 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Tag Ports During Bulk Creation" alias="tag-ports-during-bulk-creation" description="Allow to tag ports during bulk creation" updated="2019-12-29T19:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.275130 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Extension tag-ports-during-bulk-creation not supported by any of loaded plugins May 18 18:33:32.275192 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: tagging.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.276278 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Tag support for resources with standard attribute: port, subnet, subnetpool, network, security_group, router, floatingip, policy, trunk, network_segment_range" alias="standard-attr-tag" description="Enables to set tag on resources with standard attribute." updated="2017-01-01T00:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.276475 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loaded extension: standard-attr-tag May 18 18:33:32.276567 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: timestamp.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.277031 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Resource timestamps" alias="standard-attr-timestamp" description="Adds created_at and updated_at fields to all Neutron resources that have Neutron standard attributes." updated="2016-09-12T10:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.277199 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loaded extension: standard-attr-timestamp May 18 18:33:32.277257 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: trunk.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.277499 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Trunk Extension" alias="trunk" description="Provides support for trunk ports" updated="2016-01-01T10:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.277644 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Extension trunk not supported by any of loaded plugins May 18 18:33:32.277749 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: trunk_details.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.278164 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Trunk port details" alias="trunk-details" description="Expose trunk port details" updated="2016-01-01T10:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.278340 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Extension trunk-details not supported by any of loaded plugins May 18 18:33:32.278445 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: uplink_status_propagation.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.278639 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Uplink status propagation extension" alias="uplink-status-propagation" description="Provides ability to propagate port uplink status." updated="2018-05-31T18:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.278782 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Extension uplink-status-propagation not supported by any of loaded plugins May 18 18:33:32.278840 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: uplink_status_propagation_updatable.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.279030 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Uplink status propagation updatable extension" alias="uplink-status-propagation-updatable" description="Provides ability to update the port uplink status." updated="2024-09-03T18:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.279159 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Extension uplink-status-propagation-updatable not supported by any of loaded plugins May 18 18:33:32.279219 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loading extension file: vlantransparent.py {{(pid=105284) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 18 18:33:32.279459 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Ext name="Vlantransparent" alias="vlan-transparent" description="Provides Vlan Transparent Networks" updated="2015-03-23T09:00:00-00:00" {{(pid=105284) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 18 18:33:32.279613 np0000174525 neutron-ovn-maintenance-worker[105284]: INFO neutron.api.extensions [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Loaded extension: vlan-transparent May 18 18:33:32.282479 np0000174525 neutron-ovn-maintenance-worker[105284]: WARNING oslo_service.backend._threading.service [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Service MaintenanceWorker is not picklable with spawn; falling back to fork. Please make the service spawn-safe to avoid this fallback. May 18 18:33:32.282894 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Full set of CONF: {{(pid=105284) _load_service_manager_options /opt/stack/data/venv/lib/python3.12/site-packages/cotyledon/oslo_config_glue.py:59}} May 18 18:33:32.282978 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] ******************************************************************************** {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2979}} May 18 18:33:32.283035 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Configuration options gathered from: {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2980}} May 18 18:33:32.283136 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] command line args: ['--config-file', '/etc/neutron/neutron.conf', '--config-file', '/etc/neutron/plugins/ml2/ml2_conf.ini'] {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2981}} May 18 18:33:32.283235 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] config files: ['/etc/neutron/neutron.conf', '/etc/neutron/plugins/ml2/ml2_conf.ini'] {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2982}} May 18 18:33:32.283294 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] ================================================================================ {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2987}} May 18 18:33:32.283439 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] agent_down_time = 75 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.283519 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] allow_automatic_dhcp_failover = True {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.283619 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] allow_bulk = True {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.283795 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] allowed_conntrack_helpers = [{'amanda': 'tcp'}, {'ftp': 'tcp'}, {'h323': 'udp'}, {'h323': 'tcp'}, {'irc': 'tcp'}, {'netbios-ns': 'udp'}, {'pptp': 'tcp'}, {'sane': 'tcp'}, {'sip': 'udp'}, {'sip': 'tcp'}, {'snmp': 'udp'}, {'tftp': 'udp'}] {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.283867 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] api_extensions_path = {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.283972 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] api_paste_config = api-paste.ini {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.284096 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] api_workers = 4 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.284295 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] auth_strategy = keystone {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.284295 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] base_mac = fa:16:3e:00:00:00 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.284429 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] bind_host = 0.0.0.0 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.284567 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] bind_port = 9696 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.284634 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] client_socket_timeout = 900 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.284766 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] config_dir = [] {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.284854 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] config_file = ['/etc/neutron/neutron.conf', '/etc/neutron/plugins/ml2/ml2_conf.ini'] {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.284945 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] config_source = [] {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.285041 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] control_exchange = neutron {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.285157 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] core_plugin = ml2 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.285264 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] debug = True {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.285370 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] default_availability_zones = [] {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.285565 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] default_log_levels = ['amqp=WARN', 'boto=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'OFPHandler=INFO', 'OfctlService=INFO', 'os_ken.base.app_manager=INFO', 'os_ken.controller.controller=INFO'] {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.285693 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] dhcp_agent_notification = False {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.285757 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] dhcp_agents_per_network = 1 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.285862 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] dhcp_lease_duration = 86400 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.285976 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] dhcp_load_type = networks {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.286070 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] dns_domain = openstacklocal {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.286161 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] dvr_base_mac = fa:16:3f:00:00:00 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.286241 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] enable_default_route_bfd = False {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.286339 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] enable_default_route_ecmp = False {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.286458 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] enable_dvr = True {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.286563 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] enable_new_agents = True {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.286652 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] enable_services_on_agents_with_admin_state_down = False {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.286720 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] enable_snat_by_default = True {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.286816 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] enable_traditional_dhcp = True {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.286895 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] external_dns_driver = None {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.286978 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] filter_validation = True {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.287075 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] global_physnet_mtu = 1430 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.287132 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] graceful_shutdown_timeout = 60 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.287286 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] host = np0000174525 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.287386 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] host_dvr_for_dhcp = True {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.287473 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] http_retries = 3 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.287561 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] instance_format = [instance: %(uuid)s] {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.287652 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] instance_uuid_format = [instance: %(uuid)s] {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.287742 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] ipam_driver = internal {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.287828 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] ipv6_pd_enabled = False {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.287908 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] l3_ha = False {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.288002 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] l3_ha_net_cidr = 169.254.192.0/18 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.288127 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] l3_ha_network_physical_name = {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.288217 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] l3_ha_network_type = {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.288298 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] log_color = False {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.288438 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] log_config_append = None {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.288546 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.288647 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] log_dir = None {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.288723 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] log_file = None {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.288779 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] log_options = True {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.288870 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] log_rotate_interval = 1 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.288965 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] log_rotate_interval_type = days {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.289053 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] log_rotation_type = none {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.289111 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.289166 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.289268 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.289371 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.289429 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.289535 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] max_allowed_address_pair = 10 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.289651 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] max_dns_nameservers = 5 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.289752 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] max_header_line = 16384 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.289842 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] max_l3_agents_per_router = 3 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.289955 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] max_logfile_count = 30 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.290047 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] max_logfile_size_mb = 200 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.290136 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] max_routes = 30 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.290213 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] max_subnet_host_routes = 20 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.290307 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] my_ip = 199.19.213.249 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.290411 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] my_ipv6 = 2604:e100:1:0:f816:3eff:fe9a:9bc2 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.290501 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] network_auto_schedule = True {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.290620 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] network_link_prefix = None {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.290731 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] network_scheduler_driver = neutron.scheduler.dhcp_agent_scheduler.WeightScheduler {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.290824 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] notify_nova_on_port_data_changes = True {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.290909 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] notify_nova_on_port_status_changes = True {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.290990 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] pagination_max_limit = -1 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.291085 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] periodic_fuzzy_delay = 5 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.291164 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] periodic_interval = 40 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.291252 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] publish_errors = False {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.291389 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] rate_limit_burst = 0 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.291481 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] rate_limit_except_level = CRITICAL {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.291561 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] rate_limit_interval = 0 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.291647 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] router_distributed = False {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.291734 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] rpc_resources_processing_step = 20 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.291845 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] rpc_response_max_timeout = 600 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.291954 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] rpc_response_timeout = 60 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.292117 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] rpc_state_report_workers = 0 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.292201 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] rpc_workers = None {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.292309 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] send_events_interval = 2 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.292436 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] service_plugins = ['ovn-router'] {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.292526 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] setproctitle = on {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.292598 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] shell_completion = None {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.292689 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] state_path = /opt/stack/data/neutron {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.292779 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] syslog_log_facility = LOG_USER {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.292866 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] tcp_keepidle = 600 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.292954 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] transport_url = **** {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.293040 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] use_journal = False {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.293124 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] use_json = False {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.293208 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] use_stderr = False {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.293298 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] use_syslog = False {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.293394 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] vlan_qinq = None {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.293478 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] vlan_transparent = None {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.293561 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] wsgi_default_pool_size = 100 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.293647 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] wsgi_keep_alive = True {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.293740 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] wsgi_log_format = %(client_ip)s "%(request_line)s" status: %(status_code)s len: %(body_length)s time: %(wall_seconds).7f {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.293824 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] wsgi_server_debug = False {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.293933 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] oslo_concurrency.disable_process_locking = False {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.294030 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] oslo_concurrency.lock_path = /opt/stack/data/neutron/lock {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.294132 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] profiler.connection_string = messaging:// {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.294225 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] profiler.enabled = False {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.294333 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] profiler.es_doc_type = notification {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.294418 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] profiler.es_scroll_size = 10000 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.294508 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] profiler.es_scroll_time = 2m {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.294595 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] profiler.filter_error_trace = False {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.294690 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] profiler.hmac_keys = **** {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.294781 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] profiler.sentinel_service_name = mymaster {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.294883 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] profiler.socket_timeout = 0.1 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.294978 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] profiler.trace_requests = False {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.295076 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] profiler.trace_sqlalchemy = False {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.295175 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] profiler_jaeger.process_tags = {} {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.295261 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] profiler_jaeger.service_name_prefix = None {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.295365 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] profiler_otlp.service_name_prefix = None {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.295457 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.295544 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] oslo_messaging_metrics.metrics_enabled = False {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.295651 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] oslo_messaging_metrics.metrics_process_name = {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.295794 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.295908 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.296066 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] oslo_policy.enforce_new_defaults = True {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.296338 np0000174525 neutron-ovn-maintenance-worker[105284]: WARNING oslo_config.cfg [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Deprecated: Option "enforce_scope" from group "oslo_policy" is deprecated for removal (This configuration was added temporarily to facilitate a smooth transition to the new RBAC. OpenStack will always enforce scope checks. This configuration option is deprecated and will be removed in the 2025.2 cycle.). Its value may be silently ignored in the future. May 18 18:33:32.296455 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] oslo_policy.enforce_scope = True {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.296552 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] oslo_policy.policy_default_rule = default {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.296693 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] oslo_policy.policy_dirs = ['policy.d'] {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.296789 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] oslo_policy.policy_file = /etc/neutron/policy.json {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.296906 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.296997 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.297083 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] oslo_policy.remote_ssl_client_crt_file = None {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.297170 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] oslo_policy.remote_ssl_client_key_file = None {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.297255 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.297362 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] oslo_policy.remote_timeout = 60.0 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.297461 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.297552 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] service_providers.service_provider = [] {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.297652 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] nova.auth_section = None {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.297749 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] nova.auth_type = password {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.297840 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] nova.auth_url = https://199.19.213.249/identity {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.297974 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] nova.cafile = None {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.298058 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] nova.certfile = None {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.298152 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] nova.collect_timing = False {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.298238 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] nova.default_domain_id = None {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.298333 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] nova.default_domain_name = None {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.298421 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] nova.domain_id = None {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.298507 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] nova.domain_name = None {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.298639 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] nova.endpoint_type = public {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.298736 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] nova.insecure = False {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.298820 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] nova.keyfile = None {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.298910 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] nova.password = **** {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.298995 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] nova.project_domain_id = None {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.299087 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] nova.project_domain_name = Default {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.299172 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] nova.project_id = None {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.299262 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] nova.project_name = service {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.299366 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] nova.region_name = RegionOne {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.299453 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] nova.split_loggers = False {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.299535 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] nova.system_scope = None {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.299621 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] nova.timeout = None {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.299702 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] nova.trust_id = None {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.299785 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] nova.user_domain_id = None {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.299872 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] nova.user_domain_name = Default {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.299954 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] nova.user_id = None {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.300083 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] nova.username = nova {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.300168 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] placement.auth_section = None {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.300261 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] placement.auth_type = password {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.300361 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] placement.cafile = None {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.300446 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] placement.certfile = None {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.300534 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] placement.collect_timing = False {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.300624 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] placement.endpoint_type = public {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.300709 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] placement.insecure = False {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.300824 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] placement.keyfile = None {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.300928 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] placement.region_name = None {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.301016 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] placement.split_loggers = False {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.301098 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] placement.timeout = None {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.301191 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] ironic.auth_section = None {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.301275 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] ironic.auth_type = None {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.301372 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] ironic.cafile = None {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.301455 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] ironic.certfile = None {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.301544 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] ironic.collect_timing = False {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.301642 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] ironic.connect_retries = None {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.301728 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] ironic.connect_retry_delay = None {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.301815 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] ironic.enable_notifications = False {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.301899 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] ironic.endpoint_override = None {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.301984 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] ironic.insecure = False {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.302067 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] ironic.interface = None {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.302151 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] ironic.keyfile = None {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.302236 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] ironic.max_version = None {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.302336 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] ironic.min_version = None {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.302418 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] ironic.region_name = None {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.302504 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] ironic.retriable_status_codes = None {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.302588 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] ironic.service_name = None {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.302671 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] ironic.service_type = None {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.302767 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] ironic.split_loggers = False {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.302865 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] ironic.status_code_retries = None {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.303000 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] ironic.status_code_retry_delay = None {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.303093 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] ironic.timeout = None {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.303180 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] ironic.valid_interfaces = None {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.303265 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] ironic.version = None {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.303373 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] cli_script.dry_run = False {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.303474 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] oslo_messaging_rabbit.amqp_auto_delete = False {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.303562 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] oslo_messaging_rabbit.amqp_durable_queues = False {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.303653 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] oslo_messaging_rabbit.conn_pool_min_size = 2 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.303739 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] oslo_messaging_rabbit.conn_pool_ttl = 1200 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.303827 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] oslo_messaging_rabbit.direct_mandatory_flag = True {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.303911 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] oslo_messaging_rabbit.enable_cancel_on_failover = False {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.304040 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] oslo_messaging_rabbit.heartbeat_in_pthread = False {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.304140 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] oslo_messaging_rabbit.heartbeat_rate = 3 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.304228 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] oslo_messaging_rabbit.heartbeat_timeout_threshold = 60 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.304334 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] oslo_messaging_rabbit.hostname = np0000174525 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.304423 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] oslo_messaging_rabbit.kombu_compression = None {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.304516 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] oslo_messaging_rabbit.kombu_failover_strategy = round-robin {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.304608 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] oslo_messaging_rabbit.kombu_missing_consumer_retry_timeout = 60 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.304697 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] oslo_messaging_rabbit.kombu_reconnect_delay = 1.0 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.304785 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] oslo_messaging_rabbit.kombu_reconnect_splay = 0.0 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.304929 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] oslo_messaging_rabbit.processname = neutron-ovn-maintenance-worker {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.305064 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] oslo_messaging_rabbit.rabbit_ha_queues = False {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.305064 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] oslo_messaging_rabbit.rabbit_interval_max = 30 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.305224 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] oslo_messaging_rabbit.rabbit_login_method = AMQPLAIN {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.305224 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] oslo_messaging_rabbit.rabbit_qos_prefetch_count = 0 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.305299 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] oslo_messaging_rabbit.rabbit_quorum_delivery_limit = 0 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.305388 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] oslo_messaging_rabbit.rabbit_quorum_max_memory_bytes = 0 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.305473 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] oslo_messaging_rabbit.rabbit_quorum_max_memory_length = 0 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.305558 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] oslo_messaging_rabbit.rabbit_quorum_queue = False {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.305646 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] oslo_messaging_rabbit.rabbit_retry_backoff = 2 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.305733 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] oslo_messaging_rabbit.rabbit_retry_interval = 1 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.305818 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] oslo_messaging_rabbit.rabbit_stream_fanout = False {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.305903 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] oslo_messaging_rabbit.rabbit_transient_queues_ttl = 1800 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.313255 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] oslo_messaging_rabbit.rabbit_transient_quorum_queue = False {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.313255 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] oslo_messaging_rabbit.rpc_conn_pool_size = 30 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.313255 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] oslo_messaging_rabbit.ssl = False {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.313255 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] oslo_messaging_rabbit.ssl_ca_file = {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.313255 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] oslo_messaging_rabbit.ssl_cert_file = {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.313255 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] oslo_messaging_rabbit.ssl_enforce_fips_mode = False {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.313255 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] oslo_messaging_rabbit.ssl_key_file = {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.313524 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] oslo_messaging_rabbit.ssl_version = {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.313524 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] oslo_messaging_rabbit.use_queue_manager = False {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.313524 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] oslo_messaging_notifications.driver = [] {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.313524 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] oslo_messaging_notifications.retry = -1 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.313524 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] oslo_messaging_notifications.topics = ['notifications'] {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.313524 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] oslo_messaging_notifications.transport_url = **** {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.313524 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] oslo_reports.file_event_handler = None {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.313751 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] oslo_reports.file_event_handler_interval = 1 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.313751 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] oslo_reports.log_dir = None {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.313751 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] oslo_versionedobjects.fatal_exception_format_errors = False {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.313751 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] QUOTAS.default_quota = -1 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.313751 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] QUOTAS.quota_driver = neutron.db.quota.driver_nolock.DbQuotaNoLockDriver {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.313751 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] QUOTAS.quota_floatingip = 50 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.313751 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] QUOTAS.quota_network = 100 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.313985 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] QUOTAS.quota_port = 500 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.313985 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] QUOTAS.quota_rbac_policy = 10 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.313985 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] QUOTAS.quota_router = 10 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.313985 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] QUOTAS.quota_security_group = 10 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.313985 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] QUOTAS.quota_security_group_rule = 100 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.313985 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] QUOTAS.quota_subnet = 100 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.313985 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] QUOTAS.track_quota_usage = True {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.314201 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] privsep.capabilities = [21, 12, 1, 2, 19] {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.314201 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] privsep.group = None {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.314201 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] privsep.helper_command = None {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.314201 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] privsep.log_daemon_traceback = False {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.314201 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] privsep.logger_name = oslo_privsep.daemon {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.314201 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] privsep.thread_pool_size = 16 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.314201 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] privsep.user = None {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.314448 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] privsep_dhcp_release.capabilities = [21, 12] {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.314448 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] privsep_dhcp_release.group = None {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.314448 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] privsep_dhcp_release.helper_command = None {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.314448 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] privsep_dhcp_release.log_daemon_traceback = False {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.314448 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] privsep_dhcp_release.logger_name = oslo_privsep.daemon {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.314448 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] privsep_dhcp_release.thread_pool_size = 16 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.314448 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] privsep_dhcp_release.user = None {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.314688 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] privsep_ovs_vsctl.capabilities = [21, 12] {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.314688 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] privsep_ovs_vsctl.group = None {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.314688 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] privsep_ovs_vsctl.helper_command = None {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.314688 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] privsep_ovs_vsctl.log_daemon_traceback = False {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.314688 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] privsep_ovs_vsctl.logger_name = oslo_privsep.daemon {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.314688 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] privsep_ovs_vsctl.thread_pool_size = 16 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.314688 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] privsep_ovs_vsctl.user = None {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.314915 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] privsep_namespace.capabilities = [21] {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.314915 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] privsep_namespace.group = None {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.314915 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] privsep_namespace.helper_command = None {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.314915 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] privsep_namespace.log_daemon_traceback = False {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.314915 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] privsep_namespace.logger_name = oslo_privsep.daemon {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.314915 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] privsep_namespace.thread_pool_size = 16 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.314915 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] privsep_namespace.user = None {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.315195 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] privsep_conntrack.capabilities = [12] {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.315195 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] privsep_conntrack.group = None {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.315195 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] privsep_conntrack.helper_command = None {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.315195 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] privsep_conntrack.log_daemon_traceback = False {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.315195 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] privsep_conntrack.logger_name = oslo_privsep.daemon {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.315195 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] privsep_conntrack.thread_pool_size = 16 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.315195 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] privsep_conntrack.user = None {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.315413 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] privsep_link.capabilities = [12, 21] {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.315413 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] privsep_link.group = None {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.315413 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] privsep_link.helper_command = None {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.315413 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] privsep_link.log_daemon_traceback = False {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.315413 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] privsep_link.logger_name = oslo_privsep.daemon {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.315413 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] privsep_link.thread_pool_size = 16 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.315413 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] privsep_link.user = None {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.315621 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] AGENT.root_helper = sudo /opt/stack/data/venv/bin/neutron-rootwrap /etc/neutron/rootwrap.conf {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.315621 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] AGENT.root_helper_daemon = sudo /opt/stack/data/venv/bin/neutron-rootwrap-daemon /etc/neutron/rootwrap.conf {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.315621 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] AGENT.use_helper_for_ns_read = True {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.315621 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] SECURITYGROUP.enable_ipset = True {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.315621 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] SECURITYGROUP.enable_security_group = True {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.315621 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] SECURITYGROUP.firewall_driver = None {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.315920 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] SECURITYGROUP.permitted_ethertypes = [] {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.315920 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] experimental.ipv6_pd_enabled = False {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.315920 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] ml2.extension_drivers = ['port_security', 'qos'] {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.315920 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] ml2.external_network_type = None {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.315920 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] ml2.mechanism_drivers = ['ovn'] {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.315920 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] ml2.overlay_ip_version = 4 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.315920 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] ml2.path_mtu = 0 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.316382 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] ml2.physical_network_mtus = {} {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.316382 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] ml2.project_network_types = ['geneve'] {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.316382 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] ml2.tunnelled_network_rp_name = rp_tunnelled {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.316382 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] ml2.type_drivers = ['local', 'flat', 'vlan', 'geneve'] {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.316382 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] ml2_type_flat.flat_networks = ['public'] {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.316382 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] ml2_type_geneve.max_header_size = 38 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.316382 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] ml2_type_geneve.vni_ranges = ['1:65536'] {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.316681 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] ml2_type_vlan.network_vlan_ranges = ['public'] {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.316681 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] network_log.burst_limit = 25 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.316681 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] network_log.local_output_log_base = None {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.316681 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] network_log.rate_limit = 100 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.316681 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] database.asyncio_connection = **** {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.316681 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] database.asyncio_slave_connection = **** {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.316681 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] database.backend = sqlalchemy {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.316992 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] database.connection = **** {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.316992 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] database.connection_debug = 0 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.316992 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] database.connection_parameters = {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.316992 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] database.connection_recycle_time = 3600 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.316992 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] database.connection_trace = False {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.316992 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] database.db_inc_retry_interval = True {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.316992 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] database.db_max_retries = 20 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.317295 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] database.db_max_retry_interval = 10 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.317295 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] database.db_retry_interval = 1 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.317295 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] database.max_overflow = 50 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.317295 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] database.max_pool_size = 5 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.317295 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] database.max_retries = 10 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.317295 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] database.mysql_sql_mode = TRADITIONAL {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.317295 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] database.mysql_wsrep_sync_wait = None {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.317638 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] database.pool_timeout = None {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.317638 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] database.retry_interval = 10 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.317638 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] database.slave_connection = **** {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.317638 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] database.sqlite_synchronous = True {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.317638 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] database.synchronous_reader = True {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.317638 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] database.use_db_reconnect = False {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.317638 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] ovn.bfd_min_rx = 1000 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.317917 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] ovn.bfd_min_tx = 100 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.317917 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] ovn.bfd_mult = 3 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.317917 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] ovn.broadcast_arps_to_all_routers = True {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.317917 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] ovn.dhcp_default_lease_time = 43200 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.317917 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] ovn.disable_ovn_dhcp_for_baremetal_ports = False {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.317917 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] ovn.dns_records_ovn_owned = False {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.317917 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] ovn.dns_servers = [] {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.318373 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] ovn.enable_distributed_floating_ip = False {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.318373 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] ovn.fdb_age_threshold = 300 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.318373 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] ovn.ha_failover_strategy = normal {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.318373 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] ovn.live_migration_activation_strategy = rarp {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.318373 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] ovn.localnet_learn_fdb = False {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.318373 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] ovn.mac_binding_age_threshold = 300 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.318373 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] ovn.neutron_sync_mode = log {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.318711 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] ovn.ovn_dhcp4_global_options = {} {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.318711 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] ovn.ovn_dhcp6_global_options = {} {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.318711 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] ovn.ovn_l3_scheduler = leastloaded {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.318711 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] ovn.ovn_metadata_enabled = True {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.318711 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] ovn.ovn_nb_ca_cert = /opt/stack/data/CA/int-ca/ca-chain.pem {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.318711 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] ovn.ovn_nb_certificate = /opt/stack/data/CA/int-ca/devstack-cert.crt {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.318711 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] ovn.ovn_nb_connection = ['ssl:199.19.213.249:6641'] {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.319059 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] ovn.ovn_nb_private_key = /opt/stack/data/CA/int-ca/private/devstack-cert.key {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.319059 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] ovn.ovn_router_indirect_snat = False {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.319059 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] ovn.ovn_sb_ca_cert = /opt/stack/data/CA/int-ca/ca-chain.pem {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.319059 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] ovn.ovn_sb_certificate = /opt/stack/data/CA/int-ca/devstack-cert.crt {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.319059 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] ovn.ovn_sb_connection = ['ssl:199.19.213.249:6642'] {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.319059 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] ovn.ovn_sb_private_key = /opt/stack/data/CA/int-ca/private/devstack-cert.key {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.319389 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] ovn.ovs_create_tap = False {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.319389 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] ovn.ovsdb_connection_timeout = 180 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.319389 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] ovn.ovsdb_log_level = INFO {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.319389 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] ovn.ovsdb_probe_interval = 60000 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.319389 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] ovn.ovsdb_retry_max_interval = 180 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.319389 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] ovn.stateless_nat_enabled = False {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.319389 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] ovn.vhost_sock_dir = /var/run/openvswitch {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.319712 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] OVS.bridge_mac_table_size = 50000 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.319712 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] OVS.igmp_flood = False {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.319712 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] OVS.igmp_flood_reports = True {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.319849 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] OVS.igmp_flood_unregistered = False {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.319849 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] OVS.igmp_snooping_enable = False {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.319942 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] OVS.ovsdb_timeout = 10 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.320084 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] ovn_nb_global.fdb_removal_limit = 0 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.320158 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] ovn_nb_global.ignore_lsp_down = False {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.320248 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] ovn_nb_global.mac_binding_removal_limit = 0 {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.320315 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG cotyledon.oslo_config_glue [None req-9a75489d-6914-46db-b03b-7df031963107 None None] ******************************************************************************** {{(pid=105284) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3013}} May 18 18:33:32.320415 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG oslo_service.backend._threading.service [None req-9a75489d-6914-46db-b03b-7df031963107 None None] Selected the multiprocessing context: and updated it in Cotyledon ServiceManager: {{(pid=105284) launch_service /opt/stack/data/venv/lib/python3.12/site-packages/oslo_service/backend/_threading/service.py:394}} May 18 18:33:32.354638 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] Full set of CONF: {{(pid=106123) _load_service_options /opt/stack/data/venv/lib/python3.12/site-packages/cotyledon/oslo_config_glue.py:65}} May 18 18:33:32.355162 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] ******************************************************************************** {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2979}} May 18 18:33:32.355162 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] Configuration options gathered from: {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2980}} May 18 18:33:32.355162 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] command line args: ['--config-file', '/etc/neutron/neutron.conf', '--config-file', '/etc/neutron/plugins/ml2/ml2_conf.ini'] {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2981}} May 18 18:33:32.355162 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] config files: ['/etc/neutron/neutron.conf', '/etc/neutron/plugins/ml2/ml2_conf.ini'] {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2982}} May 18 18:33:32.355162 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] ================================================================================ {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2987}} May 18 18:33:32.355162 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] agent_down_time = 75 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.355549 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] allow_automatic_dhcp_failover = True {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.355549 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] allow_bulk = True {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.355549 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] allowed_conntrack_helpers = [{'amanda': 'tcp'}, {'ftp': 'tcp'}, {'h323': 'udp'}, {'h323': 'tcp'}, {'irc': 'tcp'}, {'netbios-ns': 'udp'}, {'pptp': 'tcp'}, {'sane': 'tcp'}, {'sip': 'udp'}, {'sip': 'tcp'}, {'snmp': 'udp'}, {'tftp': 'udp'}] {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.355683 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] api_extensions_path = {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.355683 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] api_paste_config = api-paste.ini {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.355810 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] api_workers = 4 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.355902 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] auth_strategy = keystone {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.356019 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] base_mac = fa:16:3e:00:00:00 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.356240 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] bind_host = 0.0.0.0 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.356357 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] bind_port = 9696 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.356442 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] client_socket_timeout = 900 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.356553 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] config_dir = [] {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.356627 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] config_file = ['/etc/neutron/neutron.conf', '/etc/neutron/plugins/ml2/ml2_conf.ini'] {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.356685 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] config_source = [] {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.356820 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] control_exchange = neutron {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.356941 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] core_plugin = ml2 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.357057 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] debug = True {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.357169 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] default_availability_zones = [] {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.357434 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] default_log_levels = ['amqp=WARN', 'boto=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'OFPHandler=INFO', 'OfctlService=INFO', 'os_ken.base.app_manager=INFO', 'os_ken.controller.controller=INFO'] {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.357576 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] dhcp_agent_notification = False {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.357700 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] dhcp_agents_per_network = 1 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.357795 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] dhcp_lease_duration = 86400 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.357928 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] dhcp_load_type = networks {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.358027 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] dns_domain = openstacklocal {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.358156 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] dvr_base_mac = fa:16:3f:00:00:00 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.358279 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] enable_default_route_bfd = False {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.358395 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] enable_default_route_ecmp = False {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.358506 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] enable_dvr = True {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.358605 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] enable_new_agents = True {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.358716 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] enable_services_on_agents_with_admin_state_down = False {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.358828 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] enable_snat_by_default = True {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.358930 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] enable_traditional_dhcp = True {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.359030 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] external_dns_driver = None {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.359085 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] fatal_deprecations = False {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.359180 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] filter_validation = True {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.359296 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] global_physnet_mtu = 1430 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.359408 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] graceful_shutdown_timeout = 60 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.359726 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] host = np0000174525 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.359850 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] host_dvr_for_dhcp = True {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.359953 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] http_retries = 3 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.360084 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] instance_format = [instance: %(uuid)s] {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.360203 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.360298 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] ipam_driver = internal {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.360417 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] ipv6_pd_enabled = False {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.360557 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] l3_ha = False {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.360676 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] l3_ha_net_cidr = 169.254.192.0/18 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.360786 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] l3_ha_network_physical_name = {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.360889 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] l3_ha_network_type = {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.360995 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] log_color = False {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.361106 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] log_config_append = None {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.361200 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.361307 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] log_dir = None {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.361428 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] log_file = None {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.361485 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] log_options = True {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.361587 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] log_rotate_interval = 1 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.361691 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] log_rotate_interval_type = days {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.361785 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] log_rotation_type = none {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.361885 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.361941 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.362051 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.366410 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.366490 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.366597 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] max_allowed_address_pair = 10 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.366721 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] max_dns_nameservers = 5 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.366848 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] max_header_line = 16384 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.366941 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] max_l3_agents_per_router = 3 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.367032 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] max_logfile_count = 30 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.367117 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] max_logfile_size_mb = 200 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.367237 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] max_routes = 30 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.367355 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] max_subnet_host_routes = 20 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.367465 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] my_ip = 199.19.213.249 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.367596 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] my_ipv6 = 2604:e100:1:0:f816:3eff:fe9a:9bc2 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.367754 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] network_auto_schedule = True {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.367877 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] network_link_prefix = None {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.368042 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] network_scheduler_driver = neutron.scheduler.dhcp_agent_scheduler.WeightScheduler {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.368148 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] notify_nova_on_port_data_changes = True {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.368304 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] notify_nova_on_port_status_changes = True {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.368441 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] pagination_max_limit = -1 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.368554 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] periodic_fuzzy_delay = 5 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.368651 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] periodic_interval = 40 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.368719 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] publish_errors = False {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.368779 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] rate_limit_burst = 0 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.368848 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] rate_limit_except_level = CRITICAL {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.368908 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] rate_limit_interval = 0 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.369342 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] router_distributed = False {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.369342 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] rpc_resources_processing_step = 20 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.369342 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] rpc_response_max_timeout = 600 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.369342 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] rpc_response_timeout = 60 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.369342 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] rpc_state_report_workers = 0 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.369575 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] rpc_workers = None {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.369643 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] send_events_interval = 2 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.369740 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] service_plugins = ['ovn-router'] {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.369843 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] setproctitle = on {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.369943 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] shell_completion = None {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.370037 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] state_path = /opt/stack/data/neutron {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.370113 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] syslog_log_facility = LOG_USER {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.370187 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] tcp_keepidle = 600 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.370273 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] transport_url = **** {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.370353 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] use_journal = False {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.370415 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] use_json = False {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.370482 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] use_stderr = False {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.370551 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] use_syslog = False {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.370605 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] vlan_qinq = None {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.370675 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] vlan_transparent = None {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.370748 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] wsgi_default_pool_size = 100 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.370826 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] wsgi_keep_alive = True {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.370920 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] wsgi_log_format = %(client_ip)s "%(request_line)s" status: %(status_code)s len: %(body_length)s time: %(wall_seconds).7f {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.370995 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] wsgi_server_debug = False {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:2995}} May 18 18:33:32.375408 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] oslo_concurrency.disable_process_locking = False {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.375555 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] oslo_concurrency.lock_path = /opt/stack/data/neutron/lock {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.375672 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] profiler.connection_string = messaging:// {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.375764 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] profiler.enabled = False {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.375922 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] profiler.es_doc_type = notification {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.375922 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] profiler.es_scroll_size = 10000 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.376042 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] profiler.es_scroll_time = 2m {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.376078 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] profiler.filter_error_trace = False {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.376184 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] profiler.hmac_keys = **** {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.376267 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] profiler.sentinel_service_name = mymaster {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.376369 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] profiler.socket_timeout = 0.1 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.376428 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] profiler.trace_requests = False {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.376495 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] profiler.trace_sqlalchemy = False {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.376588 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] profiler_jaeger.process_tags = {} {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.376650 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] profiler_jaeger.service_name_prefix = None {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.376715 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] profiler_otlp.service_name_prefix = None {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.376812 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.376888 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.376963 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_metrics.metrics_process_name = {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.377057 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.377128 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.377261 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] oslo_policy.enforce_new_defaults = True {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.377309 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] oslo_policy.enforce_scope = True {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.377376 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] oslo_policy.policy_default_rule = default {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.377420 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.377459 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] oslo_policy.policy_file = /etc/neutron/policy.json {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.377509 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.377566 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.377609 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.377656 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.377700 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.377740 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] oslo_policy.remote_timeout = 60.0 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.377803 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.378035 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] service_providers.service_provider = [] {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.378109 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] nova.auth_section = None {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.378154 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] nova.auth_type = password {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.378187 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] nova.auth_url = https://199.19.213.249/identity {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.378241 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] nova.cafile = None {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.378295 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] nova.certfile = None {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.378379 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] nova.collect_timing = False {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.378426 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] nova.default_domain_id = None {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.378466 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] nova.default_domain_name = None {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.378507 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] nova.domain_id = None {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.378550 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] nova.domain_name = None {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.378594 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] nova.endpoint_type = public {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.378653 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] nova.insecure = False {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.378694 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] nova.keyfile = None {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.378743 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] nova.password = **** {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.378781 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] nova.project_domain_id = None {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.378815 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] nova.project_domain_name = Default {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.379035 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] nova.project_id = None {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.379114 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] nova.project_name = service {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.379169 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] nova.region_name = RegionOne {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.379208 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] nova.split_loggers = False {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.379249 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] nova.system_scope = None {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.379291 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] nova.timeout = None {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.379358 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] nova.trust_id = None {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.379398 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] nova.user_domain_id = None {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.379446 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] nova.user_domain_name = Default {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.379487 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] nova.user_id = None {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.379523 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] nova.username = nova {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.379572 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] placement.auth_section = None {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.379610 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] placement.auth_type = password {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.379648 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] placement.cafile = None {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.379729 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] placement.certfile = None {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.379782 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] placement.collect_timing = False {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.379825 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] placement.endpoint_type = public {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.379868 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] placement.insecure = False {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.379909 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] placement.keyfile = None {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.379947 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] placement.region_name = None {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.380013 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] placement.split_loggers = False {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.380085 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] placement.timeout = None {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.380144 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] ironic.auth_section = None {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.380186 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] ironic.auth_type = None {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.380225 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] ironic.cafile = None {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.380267 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] ironic.certfile = None {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.380302 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] ironic.collect_timing = False {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.380356 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] ironic.connect_retries = None {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.380395 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] ironic.connect_retry_delay = None {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.380482 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] ironic.enable_notifications = False {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.380525 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] ironic.endpoint_override = None {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.380561 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] ironic.insecure = False {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.380608 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] ironic.interface = None {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.380646 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] ironic.keyfile = None {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.380688 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] ironic.max_version = None {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.380725 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] ironic.min_version = None {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.380763 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] ironic.region_name = None {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.380830 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] ironic.retriable_status_codes = None {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.380868 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] ironic.service_name = None {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.380914 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] ironic.service_type = None {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.380952 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] ironic.split_loggers = False {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.380989 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] ironic.status_code_retries = None {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.381038 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] ironic.status_code_retry_delay = None {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.381085 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] ironic.timeout = None {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.381121 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] ironic.valid_interfaces = None {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.381162 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] ironic.version = None {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.381202 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] cli_script.dry_run = False {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.381297 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.amqp_auto_delete = False {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.381362 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.amqp_durable_queues = False {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.381402 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.conn_pool_min_size = 2 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.381463 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.conn_pool_ttl = 1200 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.381509 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.direct_mandatory_flag = True {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.381546 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.enable_cancel_on_failover = False {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.381578 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.heartbeat_in_pthread = False {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.381626 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.heartbeat_rate = 3 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.381662 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.heartbeat_timeout_threshold = 60 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.381716 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.hostname = np0000174525 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.381749 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.kombu_compression = None {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.381791 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.kombu_failover_strategy = round-robin {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.381831 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.kombu_missing_consumer_retry_timeout = 60 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.381882 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.kombu_reconnect_delay = 1.0 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.381920 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.kombu_reconnect_splay = 0.0 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.381971 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.processname = neutron-ovn-maintenance-worker {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.382002 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_ha_queues = False {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.382050 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_interval_max = 30 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.382082 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_login_method = AMQPLAIN {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.382117 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_qos_prefetch_count = 0 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.382161 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_quorum_delivery_limit = 0 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.382191 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_quorum_max_memory_bytes = 0 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.382247 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_quorum_max_memory_length = 0 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.382279 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_quorum_queue = False {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.382310 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_retry_backoff = 2 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.382372 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_retry_interval = 1 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.382426 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_stream_fanout = False {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.382470 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_transient_queues_ttl = 1800 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.382501 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_transient_quorum_queue = False {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.382550 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rpc_conn_pool_size = 30 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.382602 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.ssl = False {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.382643 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.ssl_ca_file = {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.382676 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.ssl_cert_file = {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.382716 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.ssl_enforce_fips_mode = False {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.382748 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.ssl_key_file = {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.382796 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.ssl_version = {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.382838 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.use_queue_manager = False {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.382901 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_notifications.driver = [] {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.382935 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_notifications.retry = -1 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.382985 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_notifications.topics = ['notifications'] {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.383021 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_notifications.transport_url = **** {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.383095 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] oslo_reports.file_event_handler = None {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.383138 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] oslo_reports.file_event_handler_interval = 1 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.383173 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] oslo_reports.log_dir = None {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.383240 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] oslo_versionedobjects.fatal_exception_format_errors = False {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.383314 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] QUOTAS.default_quota = -1 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.383379 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] QUOTAS.quota_driver = neutron.db.quota.driver_nolock.DbQuotaNoLockDriver {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.383470 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] QUOTAS.quota_floatingip = 50 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.383516 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] QUOTAS.quota_network = 100 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.383573 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] QUOTAS.quota_port = 500 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.383613 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] QUOTAS.quota_rbac_policy = 10 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.383660 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] QUOTAS.quota_router = 10 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.383705 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] QUOTAS.quota_security_group = 10 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.383739 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] QUOTAS.quota_security_group_rule = 100 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.383791 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] QUOTAS.quota_subnet = 100 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.383824 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] QUOTAS.track_quota_usage = True {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.383895 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] privsep.capabilities = [21, 12, 1, 2, 19] {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.383945 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] privsep.group = None {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.384020 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] privsep.helper_command = None {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.384069 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] privsep.log_daemon_traceback = False {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.384108 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] privsep.logger_name = oslo_privsep.daemon {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.384146 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] privsep.thread_pool_size = 16 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.384207 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] privsep.user = None {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.384265 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] privsep_dhcp_release.capabilities = [21, 12] {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.384298 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] privsep_dhcp_release.group = None {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.384348 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] privsep_dhcp_release.helper_command = None {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.384404 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] privsep_dhcp_release.log_daemon_traceback = False {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.384448 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] privsep_dhcp_release.logger_name = oslo_privsep.daemon {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.384481 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] privsep_dhcp_release.thread_pool_size = 16 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.384514 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] privsep_dhcp_release.user = None {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.384545 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] privsep_ovs_vsctl.capabilities = [21, 12] {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.384582 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] privsep_ovs_vsctl.group = None {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.384630 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] privsep_ovs_vsctl.helper_command = None {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.384669 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] privsep_ovs_vsctl.log_daemon_traceback = False {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.384705 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] privsep_ovs_vsctl.logger_name = oslo_privsep.daemon {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.384739 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] privsep_ovs_vsctl.thread_pool_size = 16 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.384776 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] privsep_ovs_vsctl.user = None {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.384841 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] privsep_namespace.capabilities = [21] {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.384876 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] privsep_namespace.group = None {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.384919 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] privsep_namespace.helper_command = None {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.384952 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] privsep_namespace.log_daemon_traceback = False {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.384989 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] privsep_namespace.logger_name = oslo_privsep.daemon {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.385036 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] privsep_namespace.thread_pool_size = 16 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.385077 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] privsep_namespace.user = None {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.385128 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] privsep_conntrack.capabilities = [12] {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.385173 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] privsep_conntrack.group = None {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.385215 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] privsep_conntrack.helper_command = None {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.385253 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] privsep_conntrack.log_daemon_traceback = False {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.385291 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] privsep_conntrack.logger_name = oslo_privsep.daemon {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.385345 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] privsep_conntrack.thread_pool_size = 16 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.385383 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] privsep_conntrack.user = None {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.385439 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] privsep_link.capabilities = [12, 21] {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.385476 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] privsep_link.group = None {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.385520 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] privsep_link.helper_command = None {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.385559 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] privsep_link.log_daemon_traceback = False {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.385609 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] privsep_link.logger_name = oslo_privsep.daemon {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.385649 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] privsep_link.thread_pool_size = 16 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.385694 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] privsep_link.user = None {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.385732 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] AGENT.root_helper = sudo /opt/stack/data/venv/bin/neutron-rootwrap /etc/neutron/rootwrap.conf {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.385802 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] AGENT.root_helper_daemon = sudo /opt/stack/data/venv/bin/neutron-rootwrap-daemon /etc/neutron/rootwrap.conf {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.385845 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] AGENT.use_helper_for_ns_read = True {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.385909 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] SECURITYGROUP.enable_ipset = True {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.385957 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] SECURITYGROUP.enable_security_group = True {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.386006 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] SECURITYGROUP.firewall_driver = None {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.386047 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] SECURITYGROUP.permitted_ethertypes = [] {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.386088 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] experimental.ipv6_pd_enabled = False {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.386153 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] ml2.extension_drivers = ['port_security', 'qos'] {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.386197 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] ml2.external_network_type = None {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.386239 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] ml2.mechanism_drivers = ['ovn'] {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.386275 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] ml2.overlay_ip_version = 4 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.386309 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] ml2.path_mtu = 0 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.386366 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] ml2.physical_network_mtus = {} {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.386593 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] ml2.project_network_types = ['geneve'] {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.386593 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] ml2.tunnelled_network_rp_name = rp_tunnelled {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.386593 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] ml2.type_drivers = ['local', 'flat', 'vlan', 'geneve'] {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.386593 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] ml2_type_flat.flat_networks = ['public'] {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.386721 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] ml2_type_geneve.max_header_size = 38 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.386721 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] ml2_type_geneve.vni_ranges = ['1:65536'] {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.386790 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] ml2_type_vlan.network_vlan_ranges = ['public'] {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.386826 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] network_log.burst_limit = 25 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.386866 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] network_log.local_output_log_base = None {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.386913 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] network_log.rate_limit = 100 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.386960 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] database.asyncio_connection = **** {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.387001 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] database.asyncio_slave_connection = **** {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.387071 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] database.backend = sqlalchemy {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.387118 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] database.connection = **** {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.387162 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] database.connection_debug = 0 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.387162 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] database.connection_parameters = {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.387228 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] database.connection_recycle_time = 3600 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.387268 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] database.connection_trace = False {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.387306 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] database.db_inc_retry_interval = True {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.387384 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] database.db_max_retries = 20 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.387384 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] database.db_max_retry_interval = 10 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.387460 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] database.db_retry_interval = 1 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.387460 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] database.max_overflow = 50 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.387528 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] database.max_pool_size = 5 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.387528 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] database.max_retries = 10 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.387595 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] database.mysql_sql_mode = TRADITIONAL {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.387634 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] database.mysql_wsrep_sync_wait = None {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.387681 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] database.pool_timeout = None {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.387729 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] database.retry_interval = 10 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.387729 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] database.slave_connection = **** {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.387794 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] database.sqlite_synchronous = True {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.387835 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] database.synchronous_reader = True {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.387905 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] database.use_db_reconnect = False {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.388029 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] ovn.bfd_min_rx = 1000 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.388085 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] ovn.bfd_min_tx = 100 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.388118 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] ovn.bfd_mult = 3 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.388159 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] ovn.broadcast_arps_to_all_routers = True {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.388193 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] ovn.dhcp_default_lease_time = 43200 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.388229 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] ovn.disable_ovn_dhcp_for_baremetal_ports = False {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.388268 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] ovn.dns_records_ovn_owned = False {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.388305 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] ovn.dns_servers = [] {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.388374 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] ovn.enable_distributed_floating_ip = False {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.388422 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] ovn.fdb_age_threshold = 300 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.388486 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] ovn.ha_failover_strategy = normal {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.388529 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] ovn.live_migration_activation_strategy = rarp {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.388570 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] ovn.localnet_learn_fdb = False {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.388667 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] ovn.mac_binding_age_threshold = 300 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.388741 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] ovn.neutron_sync_mode = log {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.388803 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_dhcp4_global_options = {} {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.388864 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_dhcp6_global_options = {} {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.388936 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_l3_scheduler = leastloaded {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.388996 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_metadata_enabled = True {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.389070 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_nb_ca_cert = /opt/stack/data/CA/int-ca/ca-chain.pem {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.389139 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_nb_certificate = /opt/stack/data/CA/int-ca/devstack-cert.crt {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.389200 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_nb_connection = ['ssl:199.19.213.249:6641'] {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.389255 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_nb_private_key = /opt/stack/data/CA/int-ca/private/devstack-cert.key {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.389306 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_router_indirect_snat = False {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.389399 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_sb_ca_cert = /opt/stack/data/CA/int-ca/ca-chain.pem {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.389449 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_sb_certificate = /opt/stack/data/CA/int-ca/devstack-cert.crt {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.389508 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_sb_connection = ['ssl:199.19.213.249:6642'] {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.389556 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_sb_private_key = /opt/stack/data/CA/int-ca/private/devstack-cert.key {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.389629 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovs_create_tap = False {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.389670 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovsdb_connection_timeout = 180 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.389734 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovsdb_log_level = INFO {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.389788 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovsdb_probe_interval = 60000 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.389847 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovsdb_retry_max_interval = 180 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.389907 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] ovn.stateless_nat_enabled = False {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.389984 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] ovn.vhost_sock_dir = /var/run/openvswitch {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.390069 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] OVS.bridge_mac_table_size = 50000 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.390220 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] OVS.igmp_flood = False {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.390220 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] OVS.igmp_flood_reports = True {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.390302 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] OVS.igmp_flood_unregistered = False {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.390378 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] OVS.igmp_snooping_enable = False {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.390419 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] OVS.ovsdb_timeout = 10 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.390535 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] ovn_nb_global.fdb_removal_limit = 0 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.390606 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] ovn_nb_global.ignore_lsp_down = False {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.390678 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] ovn_nb_global.mac_binding_removal_limit = 0 {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3006}} May 18 18:33:32.390738 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon.oslo_config_glue [-] ******************************************************************************** {{(pid=106123) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3013}} May 18 18:33:32.390987 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG cotyledon._service_worker [-] Run service ServiceWrapper(0) [106123] {{(pid=106123) wait_forever /opt/stack/data/venv/lib/python3.12/site-packages/cotyledon/_service_worker.py:218}} May 18 18:33:32.397067 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron_lib.callbacks.manager [-] Publish callbacks ['neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver.OVNMechanismDriver.post_fork_initialize-1026903', 'neutron.services.ovn_l3.plugin.OVNL3RouterPlugin._post_fork_initialize-14412'] for process (None), after_init {{(pid=106123) _notify_loop /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:184}} May 18 18:33:32.398560 np0000174525 neutron-ovn-maintenance-worker[106123]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Getting OvsdbNbOvnIdl for MaintenanceWorker with retry May 18 18:33:32.484836 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl [-] Created lookup_table index Logical_Switch.name {{(pid=106123) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:87}} May 18 18:33:32.484981 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl [-] Created lookup_table index Logical_Router.name {{(pid=106123) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:87}} May 18 18:33:32.485019 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl [-] Created lookup_table index Load_Balancer.name {{(pid=106123) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:87}} May 18 18:33:32.485125 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Copp.name {{(pid=106123) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} May 18 18:33:32.485198 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Meter.name {{(pid=106123) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} May 18 18:33:32.485315 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Load_Balancer_Group.name {{(pid=106123) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} May 18 18:33:32.485447 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index HA_Chassis_Group.name {{(pid=106123) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} May 18 18:33:32.485551 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Address_Set.name {{(pid=106123) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} May 18 18:33:32.485646 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Connection.target {{(pid=106123) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} May 18 18:33:32.485727 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Gateway_Chassis.name {{(pid=106123) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} May 18 18:33:32.485796 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Mirror.name {{(pid=106123) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} May 18 18:33:32.485857 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Chassis_Template_Var.chassis {{(pid=106123) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} May 18 18:33:32.485932 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Logical_Router_Port.name {{(pid=106123) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} May 18 18:33:32.486001 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Port_Group.name {{(pid=106123) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} May 18 18:33:32.486060 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Logical_Switch_Port.name {{(pid=106123) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} May 18 18:33:32.489596 np0000174525 neutron-ovn-maintenance-worker[106123]: INFO ovsdbapp.backend.ovs_idl.vlog [-] ssl:199.19.213.249:6641: connecting... May 18 18:33:32.495970 np0000174525 neutron-ovn-maintenance-worker[106123]: INFO ovsdbapp.backend.ovs_idl.vlog [-] ssl:199.19.213.249:6641: connected May 18 18:33:32.519349 np0000174525 neutron-ovn-maintenance-worker[106123]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Getting OvsdbSbOvnIdl for MaintenanceWorker with retry May 18 18:33:32.551467 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl [-] Created lookup_table index Chassis.name {{(pid=106123) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:87}} May 18 18:33:32.551467 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl [-] Created lookup_table index Port_Binding.logical_port {{(pid=106123) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:87}} May 18 18:33:32.551467 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Chassis_Private.name {{(pid=106123) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} May 18 18:33:32.551720 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Datapath_Binding.tunnel_key {{(pid=106123) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} May 18 18:33:32.553757 np0000174525 neutron-ovn-maintenance-worker[106123]: INFO ovsdbapp.backend.ovs_idl.vlog [-] ssl:199.19.213.249:6642: connecting... May 18 18:33:32.558309 np0000174525 neutron-ovn-maintenance-worker[106123]: INFO ovsdbapp.backend.ovs_idl.vlog [-] ssl:199.19.213.249:6642: connected May 18 18:33:32.609705 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > floatingip precommit_update 55550000, False {{(pid=106123) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.609940 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > floatingip precommit_delete 55550000, False {{(pid=106123) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.609940 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > floatingip before_create 55550000, False {{(pid=106123) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.609940 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > floatingip before_update 55550000, False {{(pid=106123) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.610194 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port after_update 55550000, False {{(pid=106123) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.610194 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port precommit_delete 55550000, False {{(pid=106123) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.610393 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron_lib.callbacks.manager [-] Publish callbacks ['neutron.services.portforwarding.drivers.ovn.driver.OVNPortForwarding.register-14805'] for port_forwarding_plugin (None), after_init {{(pid=106123) _notify_loop /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:184}} May 18 18:33:32.610527 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port_forwarding after_create 55550000, False {{(pid=106123) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.610527 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > router_interface after_create 55550000, False {{(pid=106123) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.610686 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port_forwarding after_update 55550000, False {{(pid=106123) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.610686 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port_forwarding after_delete 55550000, False {{(pid=106123) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.610686 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > router_interface after_delete 55550000, False {{(pid=106123) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.610894 np0000174525 neutron-ovn-maintenance-worker[106123]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.extensions.qos [-] Starting OVNClientQosExtension May 18 18:33:32.610984 np0000174525 neutron-ovn-maintenance-worker[106123]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.extensions.placement [-] Starting OVNClientPlacementExtension May 18 18:33:32.611453 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > segment before_delete 55550000, False {{(pid=106123) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.611587 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > subnet precommit_delete_associations 55550000, False {{(pid=106123) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.614906 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > segment_host_mapping after_create 55550000, False {{(pid=106123) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.614997 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > segment_host_mapping after_delete 55550000, False {{(pid=106123) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.614997 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port after_create 55550000, False {{(pid=106123) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.615111 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port after_delete 55550000, False {{(pid=106123) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.615189 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port after_update 55550000, False {{(pid=106123) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.615189 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > segment after_delete 55550000, False {{(pid=106123) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.615345 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > subnet after_create 55550000, False {{(pid=106123) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.615345 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > subnet after_delete 55550000, False {{(pid=106123) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.615458 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > subnet after_update 55550000, False {{(pid=106123) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.678208 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > subnet after_create 55550000, False {{(pid=106123) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.678208 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > subnet after_delete 55550000, False {{(pid=106123) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.678208 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > subnet before_create 55550000, False {{(pid=106123) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.678505 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > subnet before_update 55550000, False {{(pid=106123) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.680980 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > security_group after_delete 55550000, False {{(pid=106123) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.681088 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port after_delete 55550000, False {{(pid=106123) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.681179 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron_lib.callbacks.manager [-] Publish callbacks ['neutron.services.logapi.drivers.ovn.driver.DriverBase._register-1850555'] for logging-plugin (None), after_init {{(pid=106123) _notify_loop /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:184}} May 18 18:33:32.681336 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > security_group_rule after_create 55550000, False {{(pid=106123) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.681407 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > security_group_rule after_update 55550000, False {{(pid=106123) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:32.681460 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > security_group_rule after_delete 55550000, False {{(pid=106123) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 18 18:33:42.056370 np0000174525 neutron-ovn-maintenance-worker[105284]: DEBUG dbcounter [-] [105284] Writing DB stats neutron:SELECT=4 {{(pid=105284) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} May 18 18:33:42.682790 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [-] OVN-Northbound DB sync process started @ 2026-05-18 18:33:42.682226 {{(pid=106123) do_sync /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:99}} May 18 18:33:42.686338 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-ecd6f7f5-28fb-4cb1-9ada-4670813e4f69 None None] OVN-NB Sync port groups started @ 2026-05-18 18:33:42.685125 {{(pid=106123) sync_port_groups /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:132}} May 18 18:33:42.708970 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG dbcounter [-] [106123] Writer thread running {{(pid=106123) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:102}} May 18 18:33:42.726760 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-ecd6f7f5-28fb-4cb1-9ada-4670813e4f69 None None] OVN-NB Sync port groups completed @ 2026-05-18 18:33:42.726459 {{(pid=106123) sync_port_groups /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:196}} May 18 18:33:42.726914 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-ecd6f7f5-28fb-4cb1-9ada-4670813e4f69 None None] OVN-NB Sync networks, ports and DHCP options started @ 2026-05-18 18:33:42.726754 {{(pid=106123) sync_networks_ports_and_dhcp_opts /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1152}} May 18 18:33:43.256725 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.services.segments.db [None req-ecd6f7f5-28fb-4cb1-9ada-4670813e4f69 None None] neutron.services.segments.plugin.Plugin method get_segments called with arguments (,) {'filters': {'network_id': ['57ad5e9e-3f36-4459-bc11-2c874585c24d'], 'is_dynamic': False}} {{(pid=106123) wrapper /opt/stack/data/venv/lib/python3.12/site-packages/oslo_log/helpers.py:77}} May 18 18:33:43.261884 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-ecd6f7f5-28fb-4cb1-9ada-4670813e4f69 None None] OVN-NB Sync metadata ports started {{(pid=106123) _sync_metadata_ports /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1097}} May 18 18:33:43.325790 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-ecd6f7f5-28fb-4cb1-9ada-4670813e4f69 None None] OVN-NB Sync metadata ports completed {{(pid=106123) _sync_metadata_ports /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1149}} May 18 18:33:43.325790 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-ecd6f7f5-28fb-4cb1-9ada-4670813e4f69 None None] OVN-NB Sync DHCP options for Neutron subnets started {{(pid=106123) _sync_subnet_dhcp_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:949}} May 18 18:33:43.398961 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-ecd6f7f5-28fb-4cb1-9ada-4670813e4f69 None None] OVN-NB Sync DHCP options for Neutron subnets completed @ 2026-05-18 18:33:43.398580 {{(pid=106123) _sync_subnet_dhcp_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1022}} May 18 18:33:43.399996 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] There are no commands to commit {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:78}} May 18 18:33:43.400289 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-ecd6f7f5-28fb-4cb1-9ada-4670813e4f69 None None] OVN-NB Sync DHCP options for Neutron ports with extra dhcp options assigned started {{(pid=106123) _sync_port_dhcp_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1027}} May 18 18:33:43.400429 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-ecd6f7f5-28fb-4cb1-9ada-4670813e4f69 None None] OVN-NB Sync DHCP options for Neutron ports with extra dhcp options assigned completed {{(pid=106123) _sync_port_dhcp_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1086}} May 18 18:33:43.400559 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-ecd6f7f5-28fb-4cb1-9ada-4670813e4f69 None None] OVN-NB Sync networks, ports and DHCP options completed @ 2026-05-18 18:33:43.400418 {{(pid=106123) sync_networks_ports_and_dhcp_opts /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1321}} May 18 18:33:43.400691 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-ecd6f7f5-28fb-4cb1-9ada-4670813e4f69 None None] OVN-NB Sync ACLs started @ 2026-05-18 18:33:43.400573 {{(pid=106123) sync_acls /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:239}} May 18 18:33:43.448140 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-ecd6f7f5-28fb-4cb1-9ada-4670813e4f69 None None] OVN-NB Sync ACLs completed @ 2026-05-18 18:33:43.447839 {{(pid=106123) sync_acls /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:431}} May 18 18:33:43.448490 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-ecd6f7f5-28fb-4cb1-9ada-4670813e4f69 None None] OVN-NB Sync routers and router ports started @ 2026-05-18 18:33:43.448111 {{(pid=106123) sync_routers_and_rports /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:736}} May 18 18:33:43.511033 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] There are no commands to commit {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:78}} May 18 18:33:43.511348 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-ecd6f7f5-28fb-4cb1-9ada-4670813e4f69 None None] OVN-NB Sync routers and router ports completed @ 2026-05-18 18:33:43.511149 {{(pid=106123) sync_routers_and_rports /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:944}} May 18 18:33:43.511465 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-ecd6f7f5-28fb-4cb1-9ada-4670813e4f69 None None] OVN-NB Sync port QoS policies started @ 2026-05-18 18:33:43.511373 {{(pid=106123) sync_port_qos_policies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1369}} May 18 18:33:43.511565 np0000174525 neutron-ovn-maintenance-worker[106123]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.extensions.qos [None req-ecd6f7f5-28fb-4cb1-9ada-4670813e4f69 None None] Starting OVNClientQosExtension May 18 18:33:43.525170 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-ecd6f7f5-28fb-4cb1-9ada-4670813e4f69 None None] OVN-NB Sync port QoS policies completed @ 2026-05-18 18:33:43.524954 {{(pid=106123) sync_port_qos_policies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1392}} May 18 18:33:43.525272 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-ecd6f7f5-28fb-4cb1-9ada-4670813e4f69 None None] OVN-NB Sync Floating IP QoS policies started @ 2026-05-18 18:33:43.525162 {{(pid=106123) sync_fip_qos_policies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1397}} May 18 18:33:43.525357 np0000174525 neutron-ovn-maintenance-worker[106123]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.extensions.qos [None req-ecd6f7f5-28fb-4cb1-9ada-4670813e4f69 None None] Starting OVNClientQosExtension May 18 18:33:43.551491 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-ecd6f7f5-28fb-4cb1-9ada-4670813e4f69 None None] OVN-NB Sync Floating IP QoS policies completed @ 2026-05-18 18:33:43.550890 {{(pid=106123) sync_fip_qos_policies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1420}} May 18 18:33:43.551771 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-ecd6f7f5-28fb-4cb1-9ada-4670813e4f69 None None] OVN-NB Sync Floating IP NAT rules started @ 2026-05-18 18:33:43.551510 {{(pid=106123) sync_fip_dnat_rules /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1425}} May 18 18:33:43.552403 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-ecd6f7f5-28fb-4cb1-9ada-4670813e4f69 None None] OVN-NB Sync Floating IP NAT rules completed @ 2026-05-18 18:33:43.552080 {{(pid=106123) sync_fip_dnat_rules /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1451}} May 18 18:33:43.552686 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-ecd6f7f5-28fb-4cb1-9ada-4670813e4f69 None None] OVN-Northbound DB sync process completed @ 2026-05-18 18:33:43.552416 {{(pid=106123) do_sync /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:112}} May 18 18:33:52.684167 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [-] OVN-Southbound DB sync process started @ 2026-05-18 18:33:52.683728 {{(pid=106123) do_sync /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1480}} May 18 18:33:52.685113 np0000174525 neutron-ovn-maintenance-worker[106123]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.extensions.qos [-] Starting OVNClientQosExtension May 18 18:33:52.687588 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-6900b341-82ff-46f2-9e66-c2c50b1c02e7 None None] OVN-SB Sync hostname and physical networks started @ 2026-05-18 18:33:52.687199 {{(pid=106123) sync_hostname_and_physical_networks /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1511}} May 18 18:33:52.690140 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.add_provnet_ext_id_to_localnet_ports {{(pid=106123) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} May 18 18:33:52.690140 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.check_baremetal_ports_dhcp_options {{(pid=106123) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} May 18 18:33:52.690140 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.check_dhcp_options_consistency {{(pid=106123) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} May 18 18:33:52.690140 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.check_fair_meter_consistency {{(pid=106123) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} May 18 18:33:52.690140 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.check_fdb_aging_settings {{(pid=106123) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} May 18 18:33:52.690675 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.check_for_igmp_snoop_support {{(pid=106123) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} May 18 18:33:52.690675 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.check_for_inconsistencies {{(pid=106123) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} May 18 18:33:52.690675 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.check_global_dhcp_opts {{(pid=106123) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} May 18 18:33:52.690675 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.check_localnet_port_has_learn_fdb {{(pid=106123) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} May 18 18:33:52.690913 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.check_network_broadcast_arps_to_all_routers {{(pid=106123) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} May 18 18:33:52.690913 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.check_provider_distributed_ports {{(pid=106123) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} May 18 18:33:52.690913 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.check_redirect_type_router_gateway_ports {{(pid=106123) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} May 18 18:33:52.691040 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip {{(pid=106123) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} May 18 18:33:52.691130 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.cleanup_old_hash_ring_nodes {{(pid=106123) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} May 18 18:33:52.693311 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.configure_nb_global {{(pid=106123) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} May 18 18:33:52.693457 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.migrate_lrp_gateway_chassis_to_ha_chassis_group {{(pid=106123) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} May 18 18:33:52.693678 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.set_fip_distributed_flag {{(pid=106123) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} May 18 18:33:52.693807 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.set_network_type_and_physnet {{(pid=106123) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} May 18 18:33:52.693929 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.set_ovn_owned_dns_option {{(pid=106123) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} May 18 18:33:52.694077 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.update_ha_failover {{(pid=106123) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} May 18 18:33:52.694144 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.update_lrouter_ports_ext_ids_name_prefix {{(pid=106123) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} May 18 18:33:52.694273 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.update_mac_aging_settings {{(pid=106123) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} May 18 18:33:52.694413 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.update_router_distributed_flag {{(pid=106123) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} May 18 18:33:52.694578 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.update_router_static_routes {{(pid=106123) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} May 18 18:33:52.694696 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.update_security_group_with_address_group {{(pid=106123) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} May 18 18:33:52.694812 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.update_virtual_port_parent_hostname {{(pid=106123) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} May 18 18:33:52.697421 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.add_provnet_ext_id_to_localnet_ports' {{(pid=106123) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} May 18 18:33:52.698270 np0000174525 neutron-ovn-maintenance-worker[106123]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver [-] Maintenance task thread has started May 18 18:33:52.698425 np0000174525 neutron-ovn-maintenance-worker[106123]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver [-] MaintenanceWorker process has finished the post initialization May 18 18:33:52.701387 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_baremetal_ports_dhcp_options' {{(pid=106123) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} May 18 18:33:52.705845 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.agent.neutron_agent [None req-6900b341-82ff-46f2-9e66-c2c50b1c02e7 None None] Agents populated in the agent cache: {{(pid=106123) populate /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/agent/neutron_agent.py:309}} May 18 18:33:52.710188 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_dhcp_options_consistency' {{(pid=106123) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} May 18 18:33:52.715219 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.agent.neutron_agent [None req-6900b341-82ff-46f2-9e66-c2c50b1c02e7 None None] - Agent type: OVN Controller Gateway agent, hostname: np0000174525, name: 0167d751-452e-4d5c-a40a-5198bde5f676 {{(pid=106123) populate /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/agent/neutron_agent.py:311}} May 18 18:33:52.717160 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fair_meter_consistency' {{(pid=106123) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} May 18 18:33:52.720347 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.agent.neutron_agent [None req-6900b341-82ff-46f2-9e66-c2c50b1c02e7 None None] - Agent type: OVN Neutron agent, hostname: np0000174525, name: 0167d751-452e-4d5c-a40a-5198bde5f676 {{(pid=106123) populate /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/agent/neutron_agent.py:311}} May 18 18:33:52.722243 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} May 18 18:33:52.723697 np0000174525 neutron-ovn-maintenance-worker[106123]: WARNING neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-6900b341-82ff-46f2-9e66-c2c50b1c02e7 None None] New host np0000174525 found in OVN SB DB, but not in Neutron. Add its SegmentHostMapping in Neutron May 18 18:33:52.724888 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_for_igmp_snoop_support' {{(pid=106123) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} May 18 18:33:52.725051 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_for_inconsistencies' {{(pid=106123) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} May 18 18:33:52.725190 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_global_dhcp_opts' {{(pid=106123) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} May 18 18:33:52.725359 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_localnet_port_has_learn_fdb' {{(pid=106123) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} May 18 18:33:52.725494 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_network_broadcast_arps_to_all_routers' {{(pid=106123) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} May 18 18:33:52.725624 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_provider_distributed_ports' {{(pid=106123) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} May 18 18:33:52.725754 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_redirect_type_router_gateway_ports' {{(pid=106123) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} May 18 18:33:52.725876 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} May 18 18:33:52.726004 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.cleanup_old_hash_ring_nodes' {{(pid=106123) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} May 18 18:33:52.726126 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.configure_nb_global' {{(pid=106123) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} May 18 18:33:52.726266 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.migrate_lrp_gateway_chassis_to_ha_chassis_group' {{(pid=106123) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} May 18 18:33:52.726408 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} May 18 18:33:52.726571 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_network_type_and_physnet' {{(pid=106123) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} May 18 18:33:52.726726 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_ovn_owned_dns_option' {{(pid=106123) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} May 18 18:33:52.726856 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} May 18 18:33:52.726980 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_lrouter_ports_ext_ids_name_prefix' {{(pid=106123) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} May 18 18:33:52.727110 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} May 18 18:33:52.727244 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_router_distributed_flag' {{(pid=106123) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} May 18 18:33:52.727402 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_router_static_routes' {{(pid=106123) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} May 18 18:33:52.727750 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_security_group_with_address_group' {{(pid=106123) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} May 18 18:33:52.727884 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_virtual_port_parent_hostname' {{(pid=106123) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} May 18 18:33:52.729060 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.add_provnet_ext_id_to_localnet_ports' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=106123) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} May 18 18:33:52.729060 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_baremetal_ports_dhcp_options {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:33:52.729174 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Sync baremetal port DHCP options with configuration. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:33:52.740786 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG oslo_concurrency.lockutils [None req-6900b341-82ff-46f2-9e66-c2c50b1c02e7 None None] Acquiring lock "update_segment_host_mapping" by "neutron.services.segments.db.update_segment_host_mapping" {{(pid=106123) inner /opt/stack/data/venv/lib/python3.12/site-packages/oslo_concurrency/lockutils.py:506}} May 18 18:33:52.741065 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG oslo_concurrency.lockutils [None req-6900b341-82ff-46f2-9e66-c2c50b1c02e7 None None] Lock "update_segment_host_mapping" acquired by "neutron.services.segments.db.update_segment_host_mapping" :: waited 0.000s {{(pid=106123) inner /opt/stack/data/venv/lib/python3.12/site-packages/oslo_concurrency/lockutils.py:519}} May 18 18:33:52.778694 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron_lib.callbacks.manager [None req-6900b341-82ff-46f2-9e66-c2c50b1c02e7 None None] Publish callbacks ['neutron.plugins.ml2.plugin.DhcpAgentSchedulerDbMixin.auto_schedule_new_network_segments-72536', 'neutron.services.segments.plugin.NovaSegmentNotifier._notify_host_addition_to_aggregate-1654620'] for segment_host_mapping (None), after_create {{(pid=106123) _notify_loop /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:184}} May 18 18:33:52.811911 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.services.segments.db [None req-6900b341-82ff-46f2-9e66-c2c50b1c02e7 None None] Segments {'cfdb2758-3efe-469e-8886-f1bbbc4f5991'} mapped to the host np0000174525 {{(pid=106123) update_segment_host_mapping /opt/stack/neutron/neutron/services/segments/db.py:249}} May 18 18:33:52.820549 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG oslo_concurrency.lockutils [None req-6900b341-82ff-46f2-9e66-c2c50b1c02e7 None None] Lock "update_segment_host_mapping" "released" by "neutron.services.segments.db.update_segment_host_mapping" :: held 0.079s {{(pid=106123) inner /opt/stack/data/venv/lib/python3.12/site-packages/oslo_concurrency/lockutils.py:538}} May 18 18:33:52.820957 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-6900b341-82ff-46f2-9e66-c2c50b1c02e7 None None] OVN-SB Sync hostname and physical networks completed @ 2026-05-18 18:33:52.820499 {{(pid=106123) sync_hostname_and_physical_networks /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1538}} May 18 18:33:52.821689 np0000174525 neutron-ovn-maintenance-worker[106123]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_baremetal_ports_dhcp_options finished in 0.092 seconds May 18 18:33:52.821947 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_baremetal_ports_dhcp_options' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=106123) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} May 18 18:33:52.822195 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_dhcp_options_consistency {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:33:52.822368 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Backfill network_id on DHCP_Options external_ids. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:33:52.881074 np0000174525 neutron-ovn-maintenance-worker[106123]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_dhcp_options_consistency finished in 0.058 seconds May 18 18:33:52.881074 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_dhcp_options_consistency' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=106123) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} May 18 18:33:52.881422 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fair_meter_consistency {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:33:52.881422 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Refresh ACL logging fair meter after configuration reload. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:33:52.881853 np0000174525 neutron-ovn-maintenance-worker[106123]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fair_meter_consistency finished in 0.000 seconds May 18 18:33:52.882010 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fair_meter_consistency' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=106123) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} May 18 18:33:52.882160 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:33:52.882301 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:33:52.886244 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:33:52.994818 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:33:52.995582 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:33:53.002400 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:33:53.002400 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:33:53.002400 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:33:53.002400 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:33:53.002400 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:33:53.002400 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:33:53.002400 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:33:53.002400 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:33:53.022025 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.125 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:33:53.022025 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:33:53.022025 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:33:53.022025 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:33:53.022025 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:33:53.022025 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:33:53.022025 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:33:53.022025 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:33:53.022025 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:33:53.022025 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:33:53.022025 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:33:53.022025 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:33:53.022722 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:33:53.022722 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:33:53.022722 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:33:53.022722 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:33:53.022722 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:33:53.022722 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:33:53.022722 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:33:53.022722 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:33:53.022722 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:33:53.022722 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:33:53.022722 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:33:53.022722 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:33:53.022722 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:33:53.023395 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:33:53.023395 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:33:53.023395 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call immediate 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:33:53.023395 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:33:53.023395 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:33:53.023395 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:33:53.023395 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:33:53.023395 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:33:53.023395 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:33:53.023395 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:33:53.023395 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:33:53.023395 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:33:53.023395 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:33:53.023395 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:33:53.023395 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:33:53.024253 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:33:53.024253 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:33:53.024253 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:33:53.024253 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:33:53.024253 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:33:53.024253 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:33:53.024253 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:33:53.024253 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:33:53.024253 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:33:53.024253 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:33:53.024253 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:33:53.024253 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:33:53.024253 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:33:53.024253 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:33:53.024253 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:33:53.024253 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:33:53.024253 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:33:53.024253 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:33:53.024253 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:33:53.025219 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:33:53.025219 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:33:53.025219 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:33:53.025219 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:33:53.025219 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:33:53.025219 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:33:53.025219 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_for_igmp_snoop_support {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:33:53.025219 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Check for IGMP snoop support. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:33:53.025219 np0000174525 neutron-ovn-maintenance-worker[106123]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_for_igmp_snoop_support finished in 0.000 seconds May 18 18:33:53.025219 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_for_igmp_snoop_support' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=106123) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} May 18 18:33:53.025796 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_for_inconsistencies {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:33:53.025796 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Checking Neutron and OVN revision consistency. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:33:53.055755 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Maintenance task: No inconsistencies found. Skipping {{(pid=106123) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:437}} May 18 18:33:53.055940 np0000174525 neutron-ovn-maintenance-worker[106123]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_for_inconsistencies finished in 0.031 seconds May 18 18:33:53.056196 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_global_dhcp_opts {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:33:53.056333 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Check global DHCP options consistency. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:33:53.056599 np0000174525 neutron-ovn-maintenance-worker[106123]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_global_dhcp_opts finished in 0.000 seconds May 18 18:33:53.056952 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_global_dhcp_opts' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=106123) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} May 18 18:33:53.057110 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_localnet_port_has_learn_fdb {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:33:53.057245 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Ensure localnet ports have learn-fdb set per configuration. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:33:53.060253 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] There are no commands to commit {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:78}} May 18 18:33:53.060960 np0000174525 neutron-ovn-maintenance-worker[106123]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_localnet_port_has_learn_fdb finished in 0.003 seconds May 18 18:33:53.061051 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_localnet_port_has_learn_fdb' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=106123) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} May 18 18:33:53.061225 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_network_broadcast_arps_to_all_routers {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:33:53.061380 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Sync broadcast-arps-to-all-routers on external networks. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:33:53.118219 np0000174525 neutron-ovn-maintenance-worker[106123]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_network_broadcast_arps_to_all_routers finished in 0.056 seconds May 18 18:33:53.118451 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_network_broadcast_arps_to_all_routers' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=106123) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} May 18 18:33:53.118670 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_provider_distributed_ports {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:33:53.118769 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Align reside-on-redirect-chassis on VLAN/FLAT distributed ports. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:33:53.259804 np0000174525 neutron-ovn-maintenance-worker[106123]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_provider_distributed_ports finished in 0.141 seconds May 18 18:33:53.260058 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_provider_distributed_ports' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=106123) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} May 18 18:33:53.260487 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_redirect_type_router_gateway_ports {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:33:53.260663 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Align router gateway redirect-type for provider VLAN/FLAT networks. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:33:53.283629 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] There are no commands to commit {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:78}} May 18 18:33:53.284300 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-6900b341-82ff-46f2-9e66-c2c50b1c02e7 None None] OVN-Southbound DB sync process completed @ 2026-05-18 18:33:53.283869 {{(pid=106123) do_sync /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1488}} May 18 18:33:53.350859 np0000174525 neutron-ovn-maintenance-worker[106123]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_redirect_type_router_gateway_ports finished in 0.090 seconds May 18 18:33:53.351012 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_redirect_type_router_gateway_ports' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=106123) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} May 18 18:33:53.351179 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:33:53.351292 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:33:53.352994 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:33:53.353424 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:33:53.354667 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:33:53.354667 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:33:53.354667 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:33:53.354667 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:33:53.354667 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:33:53.354667 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:33:53.354667 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:33:53.354667 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:33:53.355652 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:33:53.355652 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:33:53.355652 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:33:53.355652 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:33:53.355652 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:33:53.355652 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:33:53.355652 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:33:53.355652 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:33:53.355652 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:33:53.355652 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:33:53.355652 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:33:53.355652 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:33:53.356243 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:33:53.356243 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:33:53.356243 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:33:53.356243 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:33:53.356243 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:33:53.356243 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:33:53.356243 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:33:53.356243 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:33:53.356243 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:33:53.356243 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:33:53.356243 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:33:53.356243 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:33:53.356243 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:33:53.358039 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:33:53.358039 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:33:53.358039 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call immediate 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:33:53.358039 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:33:53.358039 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:33:53.358039 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:33:53.358039 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:33:53.358039 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:33:53.358039 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:33:53.358039 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:33:53.358039 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:33:53.358039 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:33:53.358039 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:33:53.358039 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:33:53.358039 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:33:53.359162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:33:53.359162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:33:53.359162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:33:53.359162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:33:53.359162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:33:53.359162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:33:53.359162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:33:53.359162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:33:53.359162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:33:53.359162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:33:53.359162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:33:53.359162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:33:53.359162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:33:53.359162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:33:53.359162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:33:53.359162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:33:53.359162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:33:53.359162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:33:53.359162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:33:53.359905 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:33:53.359905 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:33:53.359905 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:33:53.359905 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:33:53.359905 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:33:53.359905 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:33:53.359905 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: cleanup_old_hash_ring_nodes {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:33:53.359905 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Purge stale hash ring nodes older than five days. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:33:53.360750 np0000174525 neutron-ovn-maintenance-worker[106123]: INFO neutron.db.ovn_hash_ring_db [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Cleaned up Hash Ring nodes older than 5 days May 18 18:33:53.360843 np0000174525 neutron-ovn-maintenance-worker[106123]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task cleanup_old_hash_ring_nodes finished in 0.004 seconds May 18 18:33:53.361006 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: configure_nb_global {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:33:53.361086 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply ovn_nb_global settings to NB_Global options. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:33:53.361305 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'ignore_lsp_down': 'false', 'fdb_removal_limit': '0', 'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:33:53.361872 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'ignore_lsp_down': 'false', 'fdb_removal_limit': '0', 'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:33:53.362341 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:33:53.362980 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:33:53.362980 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:33:53.362980 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:33:53.362980 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:33:53.362980 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:33:53.362980 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:33:53.362980 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:33:53.362980 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:33:53.365579 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:33:53.365579 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:33:53.365579 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:33:53.365579 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:33:53.365579 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:33:53.365579 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:33:53.365579 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:33:53.365579 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:33:53.365579 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:33:53.365579 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:33:53.365579 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:33:53.365579 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:33:53.365579 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:33:53.365579 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:33:53.365579 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:33:53.366104 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:33:53.366104 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:33:53.366104 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:33:53.366104 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:33:53.366104 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:33:53.366104 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:33:53.366104 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:33:53.366104 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:33:53.366104 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:33:53.366432 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task configure_nb_global failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:33:53.366432 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:33:53.366432 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:33:53.366432 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:33:53.366432 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:33:53.366432 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1038, in configure_nb_global May 18 18:33:53.366432 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:33:53.366432 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:33:53.366432 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:33:53.366432 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:33:53.366432 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:33:53.366732 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:33:53.366732 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:33:53.366732 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:33:53.366732 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:33:53.366732 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:33:53.366732 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:33:53.366732 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:33:53.366732 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:33:53.366732 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:33:53.366732 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:33:53.366732 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:33:53.366732 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:33:53.366732 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:33:53.366732 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:33:53.367077 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:33:53.367077 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:33:53.367077 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:33:53.367077 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:33:53.367077 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call immediate 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.configure_nb_global' (it runs every 86400.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:33:53.367077 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:33:53.367077 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:33:53.367077 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:33:53.367077 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:33:53.367077 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:33:53.367077 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:33:53.367077 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:33:53.367077 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:33:53.367077 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:33:53.367586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:33:53.367586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:33:53.367586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:33:53.367586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:33:53.367586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:33:53.367586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:33:53.367586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1038, in configure_nb_global May 18 18:33:53.367586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:33:53.367586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:33:53.367586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:33:53.367586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:33:53.367586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:33:53.367586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:33:53.367586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:33:53.367586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:33:53.367586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:33:53.367586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:33:53.367586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:33:53.367586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:33:53.368088 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:33:53.368088 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:33:53.368088 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:33:53.368088 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:33:53.368088 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:33:53.368088 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:33:53.368088 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:33:53.368088 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:33:53.368088 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:33:53.368088 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:33:53.368088 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:33:53.368088 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] There are no commands to commit {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:78}} May 18 18:33:53.368088 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.migrate_lrp_gateway_chassis_to_ha_chassis_group' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=106123) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} May 18 18:33:53.368541 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:33:53.368541 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:33:53.368541 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:33:53.368694 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:33:53.368904 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:33:53.369493 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:33:53.369493 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:33:53.369493 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:33:53.369493 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:33:53.369493 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:33:53.369493 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:33:53.369493 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:33:53.369493 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:33:53.370056 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:33:53.370056 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:33:53.370056 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:33:53.370056 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:33:53.370056 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:33:53.370056 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:33:53.370056 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:33:53.370056 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:33:53.370056 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:33:53.370056 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:33:53.370056 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:33:53.370056 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:33:53.370056 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:33:53.370056 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:33:53.370056 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:33:53.370500 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:33:53.370500 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:33:53.370500 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:33:53.370500 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:33:53.370500 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:33:53.370500 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:33:53.370500 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:33:53.370500 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:33:53.370500 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:33:53.370741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:33:53.370741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:33:53.370741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:33:53.370741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:33:53.370741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:33:53.370741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:33:53.370741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:33:53.370741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:33:53.370741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:33:53.370741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:33:53.370741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:33:53.370741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:33:53.371065 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:33:53.371065 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:33:53.371065 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:33:53.371065 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:33:53.371065 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:33:53.371065 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:33:53.371065 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:33:53.371065 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:33:53.371065 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:33:53.371065 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:33:53.371065 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:33:53.371065 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:33:53.371065 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:33:53.371065 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:33:53.371454 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:33:53.371454 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:33:53.371454 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:33:53.371454 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:33:53.371454 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call immediate 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:33:53.371454 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:33:53.371454 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:33:53.371454 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:33:53.371454 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:33:53.371454 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:33:53.371454 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:33:53.371454 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:33:53.371454 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:33:53.371454 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:33:53.371821 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:33:53.371821 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:33:53.371821 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:33:53.371821 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:33:53.371821 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:33:53.371821 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:33:53.371821 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:33:53.371821 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:33:53.371821 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:33:53.371821 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:33:53.371821 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:33:53.371821 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:33:53.371821 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:33:53.371821 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:33:53.371821 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:33:53.371821 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:33:53.371821 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:33:53.371821 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:33:53.371821 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:33:53.372292 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:33:53.372292 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:33:53.372292 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:33:53.372292 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:33:53.372292 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:33:53.372292 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:33:53.372292 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:33:53.372292 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:33:53.372292 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:33:53.372292 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:33:53.372292 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:33:53.372292 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:33:53.372292 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_network_type_and_physnet {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:33:53.372292 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set network type and physnet external_ids on logical switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:33:53.376694 np0000174525 neutron-ovn-maintenance-worker[106123]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_network_type_and_physnet finished in 0.005 seconds May 18 18:33:53.376776 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_network_type_and_physnet' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=106123) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} May 18 18:33:53.376915 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_ovn_owned_dns_option {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:33:53.376989 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply ovn_owned option on DNS records per configuration. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:33:53.377181 np0000174525 neutron-ovn-maintenance-worker[106123]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_ovn_owned_dns_option finished in 0.000 seconds May 18 18:33:53.377276 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_ovn_owned_dns_option' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=106123) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} May 18 18:33:53.377410 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:33:53.377472 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:33:53.377704 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:33:53.378177 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:33:53.378495 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:33:53.379158 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:33:53.379158 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:33:53.379158 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:33:53.379158 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:33:53.379158 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:33:53.379158 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:33:53.379158 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:33:53.379158 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:33:53.380155 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:33:53.380155 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:33:53.380155 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:33:53.380155 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:33:53.380155 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:33:53.380155 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:33:53.380155 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:33:53.380155 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:33:53.380155 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:33:53.380155 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:33:53.380155 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:33:53.380155 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:33:53.380155 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:33:53.380155 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:33:53.380155 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:33:53.380624 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:33:53.380624 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:33:53.380624 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:33:53.380624 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:33:53.380624 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:33:53.380624 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:33:53.380624 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:33:53.380624 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:33:53.380624 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:33:53.380894 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:33:53.380894 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:33:53.380894 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:33:53.380894 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:33:53.380894 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:33:53.380894 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:33:53.380894 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:33:53.380894 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:33:53.380894 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:33:53.380894 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:33:53.380894 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:33:53.381220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:33:53.381220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:33:53.381220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:33:53.381220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:33:53.381220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:33:53.381220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:33:53.381220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:33:53.381220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:33:53.381220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:33:53.381220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:33:53.381220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:33:53.381220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:33:53.381220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:33:53.381220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:33:53.381608 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:33:53.381608 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:33:53.381608 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:33:53.381608 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:33:53.381608 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call immediate 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:33:53.381608 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:33:53.381608 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:33:53.381608 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:33:53.381608 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:33:53.381608 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:33:53.381608 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:33:53.381608 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:33:53.381608 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:33:53.381608 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:33:53.382039 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:33:53.382039 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:33:53.382039 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:33:53.382039 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:33:53.382039 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:33:53.382039 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:33:53.382039 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:33:53.382039 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:33:53.382039 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:33:53.382039 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:33:53.382039 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:33:53.382039 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:33:53.382039 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:33:53.382039 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:33:53.382039 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:33:53.382039 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:33:53.382039 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:33:53.382039 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:33:53.382039 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:33:53.382588 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:33:53.382588 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:33:53.382588 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:33:53.382588 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:33:53.382588 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:33:53.382588 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:33:53.382588 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:33:53.382588 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:33:53.382588 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:33:53.382588 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:33:53.382588 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:33:53.382588 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_lrouter_ports_ext_ids_name_prefix {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:33:53.382588 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: ('Check missing prefix router_name', 'in external_ids of LRPs.') {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:33:53.382952 np0000174525 neutron-ovn-maintenance-worker[106123]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_lrouter_ports_ext_ids_name_prefix finished in 0.000 seconds May 18 18:33:53.382952 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_lrouter_ports_ext_ids_name_prefix' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=106123) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} May 18 18:33:53.382952 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:33:53.382952 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:33:53.382952 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:33:53.383771 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:33:53.384235 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:33:53.384480 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:33:53.385345 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:33:53.385345 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:33:53.385345 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:33:53.385345 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:33:53.385345 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:33:53.385345 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:33:53.385345 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:33:53.385345 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:33:53.385966 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:33:53.385966 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:33:53.385966 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:33:53.385966 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:33:53.385966 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:33:53.385966 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:33:53.385966 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:33:53.385966 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:33:53.385966 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:33:53.385966 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:33:53.385966 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:33:53.385966 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:33:53.386416 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:33:53.386416 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:33:53.386416 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:33:53.386416 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:33:53.386416 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:33:53.386416 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:33:53.386416 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:33:53.386416 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:33:53.386416 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:33:53.386416 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:33:53.386416 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:33:53.386416 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:33:53.386416 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:33:53.386765 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:33:53.386765 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:33:53.386765 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call immediate 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:33:53.386765 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:33:53.386765 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:33:53.386765 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:33:53.386765 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:33:53.386765 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:33:53.386765 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:33:53.386765 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:33:53.386765 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:33:53.386765 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:33:53.386765 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:33:53.386765 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:33:53.386765 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:33:53.387271 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:33:53.387271 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:33:53.387271 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:33:53.387271 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:33:53.387271 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:33:53.387271 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:33:53.387271 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:33:53.387271 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:33:53.387271 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:33:53.387271 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:33:53.387271 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:33:53.387271 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:33:53.387271 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:33:53.387271 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:33:53.387271 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:33:53.387271 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:33:53.387271 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:33:53.387271 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:33:53.387271 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:33:53.387997 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:33:53.387997 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:33:53.387997 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:33:53.387997 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:33:53.387997 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:33:53.387997 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:33:53.387997 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_router_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:33:53.387997 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Sync router.distributed with distributed floating IP config. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:33:53.390523 np0000174525 neutron-ovn-maintenance-worker[106123]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_router_distributed_flag finished in 0.004 seconds May 18 18:33:53.390592 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_router_distributed_flag' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=106123) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} May 18 18:33:53.390727 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_router_static_routes {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:33:53.390795 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Mark Neutron-owned static routes in OVN external_ids. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:33:53.391489 np0000174525 neutron-ovn-maintenance-worker[106123]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_router_static_routes finished in 0.001 seconds May 18 18:33:53.391551 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_router_static_routes' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=106123) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} May 18 18:33:53.391682 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_security_group_with_address_group {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:33:53.391764 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Repair Address_Sets and ACLs for address group rules. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:33:53.401347 np0000174525 neutron-ovn-maintenance-worker[106123]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_security_group_with_address_group finished in 0.009 seconds May 18 18:33:53.401507 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_security_group_with_address_group' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=106123) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} May 18 18:33:53.405468 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_virtual_port_parent_hostname' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=106123) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} May 18 18:33:57.882732 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:33:57.884363 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:33:57.884586 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:33:57.885938 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:33:57.916879 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:33:57.917430 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:33:57.918859 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:33:57.918859 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:33:57.918859 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:33:57.918859 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:33:57.918859 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:33:57.918859 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:33:57.918859 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:33:57.918859 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:33:57.919729 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.034 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:33:57.919729 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:33:57.919729 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:33:57.919729 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:33:57.919729 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:33:57.919729 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:33:57.919729 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:33:57.919729 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:33:57.919729 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:33:57.919729 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:33:57.919729 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:33:57.919729 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:33:57.920120 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:33:57.920120 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:33:57.920120 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:33:57.920120 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:33:57.920120 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:33:57.920120 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:33:57.920120 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:33:57.920120 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:33:57.920120 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:33:57.920120 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:33:57.920120 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:33:57.920120 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:33:57.920120 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:33:57.920502 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:33:57.920502 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:33:57.920612 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:33:57.920612 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:33:57.920612 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:33:57.920612 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:33:57.920612 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:33:57.920612 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:33:57.920612 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:33:57.920612 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:33:57.920612 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:33:57.920612 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:33:57.920612 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:33:57.920612 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:33:57.920612 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:33:57.920612 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:33:57.920612 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:33:57.920612 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:33:57.921116 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:33:57.921116 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:33:57.921116 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:33:57.921116 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:33:57.921116 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:33:57.921116 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:33:57.921116 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:33:57.921116 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:33:57.921116 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:33:57.921116 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:33:57.921116 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:33:57.921116 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:33:57.921116 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:33:57.921116 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:33:57.921116 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:33:57.921116 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:33:57.921116 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:33:57.921116 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:33:57.921116 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:33:57.921596 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:33:57.921596 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:33:57.921596 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:33:58.351746 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:33:58.352652 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:33:58.352853 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:33:58.354193 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:33:58.354612 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:33:58.355624 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:33:58.355624 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:33:58.355624 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:33:58.355624 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:33:58.355624 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:33:58.355624 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:33:58.355624 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:33:58.355624 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:33:58.357294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:33:58.357294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:33:58.357294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:33:58.357294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:33:58.357294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:33:58.357294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:33:58.357294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:33:58.357294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:33:58.357294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:33:58.357294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:33:58.357294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:33:58.357294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:33:58.357960 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:33:58.357960 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:33:58.357960 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:33:58.357960 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:33:58.357960 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:33:58.357960 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:33:58.357960 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:33:58.357960 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:33:58.357960 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:33:58.357960 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:33:58.357960 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:33:58.357960 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:33:58.357960 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:33:58.358645 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:33:58.358645 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:33:58.358967 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:33:58.358967 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:33:58.358967 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:33:58.358967 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:33:58.358967 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:33:58.358967 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:33:58.358967 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:33:58.358967 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:33:58.358967 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:33:58.358967 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:33:58.358967 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:33:58.358967 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:33:58.358967 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:33:58.358967 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:33:58.358967 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:33:58.358967 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:33:58.359866 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:33:58.359866 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:33:58.359866 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:33:58.359866 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:33:58.359866 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:33:58.359866 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:33:58.359866 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:33:58.359866 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:33:58.359866 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:33:58.359866 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:33:58.359866 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:33:58.359866 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:33:58.359866 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:33:58.359866 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:33:58.359866 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:33:58.359866 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:33:58.359866 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:33:58.359866 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:33:58.359866 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:33:58.360867 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:33:58.360867 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:33:58.360867 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:33:58.368138 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:33:58.368556 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:33:58.368687 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:33:58.368851 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:33:58.369506 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:33:58.369933 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:33:58.371010 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:33:58.371010 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:33:58.371010 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:33:58.371010 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:33:58.371010 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:33:58.371010 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:33:58.371010 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:33:58.371010 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:33:58.372576 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:33:58.372576 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:33:58.372576 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:33:58.372576 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:33:58.372576 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:33:58.372576 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:33:58.372576 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:33:58.372576 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:33:58.372576 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:33:58.372576 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:33:58.372576 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:33:58.372576 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:33:58.372576 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:33:58.372576 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:33:58.372576 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:33:58.373318 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:33:58.373318 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:33:58.373318 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:33:58.373318 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:33:58.373318 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:33:58.373318 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:33:58.373318 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:33:58.373318 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:33:58.373318 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:33:58.373884 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:33:58.373884 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:33:58.373884 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:33:58.373884 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:33:58.373884 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:33:58.373884 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:33:58.373884 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:33:58.373884 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:33:58.373884 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:33:58.373884 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:33:58.373884 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:33:58.373884 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:33:58.374502 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:33:58.374502 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:33:58.374502 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:33:58.374502 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:33:58.374502 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:33:58.374502 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:33:58.374502 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:33:58.374502 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:33:58.374502 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:33:58.374502 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:33:58.374502 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:33:58.374502 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:33:58.374502 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:33:58.374502 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:33:58.375241 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:33:58.375241 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:33:58.375241 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:33:58.375241 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:33:58.375523 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:33:58.375523 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:33:58.375523 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:33:58.375523 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:33:58.375523 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:33:58.375523 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:33:58.375523 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:33:58.375523 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:33:58.375523 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:33:58.375523 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:33:58.375523 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:33:58.375523 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:33:58.375523 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:33:58.375523 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:33:58.375523 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:33:58.375523 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:33:58.376513 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:33:58.376513 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:33:58.376513 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:33:58.376513 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:33:58.376513 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:33:58.376513 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:33:58.376513 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:33:58.376513 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:33:58.376513 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:33:58.376513 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:33:58.376513 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:33:58.376513 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:33:58.376513 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:33:58.376513 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:33:58.376513 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:33:58.376513 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:33:58.376513 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:33:58.376513 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:33:58.377314 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:33:58.377314 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:33:58.377314 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:33:58.377314 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:33:58.377314 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:33:58.377314 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:33:58.377314 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:33:58.377676 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:33:58.378071 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:33:58.378198 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:33:58.378294 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:33:58.378916 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:33:58.379283 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:33:58.380091 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:33:58.380091 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:33:58.380091 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:33:58.380091 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:33:58.380091 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:33:58.380091 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:33:58.380091 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:33:58.380091 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:33:58.380915 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:33:58.380915 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:33:58.380915 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:33:58.380915 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:33:58.380915 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:33:58.380915 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:33:58.380915 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:33:58.380915 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:33:58.380915 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:33:58.380915 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:33:58.380915 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:33:58.380915 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:33:58.380915 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:33:58.380915 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:33:58.380915 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:33:58.381343 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:33:58.381343 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:33:58.381343 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:33:58.381343 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:33:58.381343 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:33:58.381343 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:33:58.381343 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:33:58.381343 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:33:58.381343 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:33:58.381883 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:33:58.381883 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:33:58.381883 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:33:58.381883 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:33:58.381883 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:33:58.381883 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:33:58.381883 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:33:58.381883 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:33:58.381883 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:33:58.381883 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:33:58.381883 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:33:58.382206 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:33:58.382206 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:33:58.382206 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:33:58.382206 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:33:58.382206 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:33:58.382206 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:33:58.382206 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:33:58.382206 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:33:58.382206 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:33:58.382206 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:33:58.382206 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:33:58.382206 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:33:58.382206 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:33:58.382206 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:33:58.382598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:33:58.382598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:33:58.382598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:33:58.382598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:33:58.383020 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:33:58.383020 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:33:58.383020 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:33:58.383020 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:33:58.383020 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:33:58.383020 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:33:58.383020 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:33:58.383020 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:33:58.383020 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:33:58.383020 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:33:58.383020 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:33:58.383020 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:33:58.383020 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:33:58.383020 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:33:58.383020 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:33:58.383020 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:33:58.383020 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:33:58.383583 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:33:58.383583 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:33:58.383583 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:33:58.383583 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:33:58.383583 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:33:58.383583 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:33:58.383583 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:33:58.383583 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:33:58.383583 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:33:58.383583 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:33:58.383583 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:33:58.383583 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:33:58.383583 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:33:58.383583 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:33:58.383583 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:33:58.383583 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:33:58.383583 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:33:58.383583 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:33:58.383583 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:33:58.384170 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:33:58.384170 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:33:58.384170 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:33:58.384170 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:33:58.384170 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:33:58.384170 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:33:58.384371 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:33:58.384485 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:33:58.386048 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:33:58.386048 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:33:58.386048 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:33:58.386235 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:33:58.386235 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:33:58.386235 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:33:58.386235 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:33:58.386235 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:33:58.386235 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:33:58.386235 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:33:58.386235 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:33:58.387191 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:33:58.387191 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:33:58.387191 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:33:58.387191 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:33:58.387191 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:33:58.387191 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:33:58.387191 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:33:58.387191 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:33:58.387191 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:33:58.387191 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:33:58.387191 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:33:58.387191 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:33:58.387597 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:33:58.387597 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:33:58.387597 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:33:58.387597 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:33:58.387597 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:33:58.387597 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:33:58.387597 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:33:58.387597 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:33:58.387597 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:33:58.387597 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:33:58.387597 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:33:58.387597 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:33:58.387597 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:33:58.387979 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:33:58.387979 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:33:58.388280 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:33:58.388280 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:33:58.388280 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:33:58.388280 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:33:58.388280 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:33:58.388280 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:33:58.388280 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:33:58.388280 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:33:58.388280 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:33:58.388280 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:33:58.388280 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:33:58.388280 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:33:58.388280 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:33:58.388280 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:33:58.388280 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:33:58.388280 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:33:58.388810 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:33:58.388810 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:33:58.388810 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:33:58.388810 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:33:58.388810 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:33:58.388810 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:33:58.388810 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:33:58.388810 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:33:58.388810 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:33:58.388810 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:33:58.388810 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:33:58.388810 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:33:58.388810 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:33:58.388810 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:33:58.388810 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:33:58.388810 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:33:58.388810 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:33:58.388810 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:33:58.388810 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:33:58.389485 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:33:58.389485 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:33:58.389485 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:34:02.885136 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:34:02.885752 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:34:02.885881 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:34:02.887468 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:34:02.917956 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:34:02.918362 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:02.918964 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:34:02.918964 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:02.918964 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:34:02.918964 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:34:02.918964 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:02.918964 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:34:02.918964 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:02.918964 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:34:02.919674 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.033 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:02.919674 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:34:02.919674 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:02.919674 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:34:02.919674 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:34:02.919674 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:34:02.919674 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:34:02.919674 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:02.919674 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:02.919674 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:02.919674 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:34:02.919674 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:02.920543 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:02.920543 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:02.920543 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:34:02.920543 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:02.920543 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:34:02.920543 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:34:02.920543 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:02.920543 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:34:02.920543 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:02.920543 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:34:02.920543 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:34:02.920543 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:02.920543 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:34:02.921116 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:02.921116 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:34:02.921116 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:02.921116 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:34:02.921116 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:34:02.921116 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:34:02.921116 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:34:02.921116 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:34:02.921116 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:02.921116 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:34:02.921116 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:34:02.921116 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:02.921116 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:34:02.921116 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:34:02.921116 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:02.921779 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:02.921779 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:34:02.921779 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:02.921779 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:34:02.921779 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:34:02.921779 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:02.921779 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:02.921779 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:02.921779 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:34:02.921779 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:02.921779 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:02.921779 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:02.921779 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:34:02.921779 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:02.921779 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:34:02.921779 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:34:02.921779 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:02.921779 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:34:02.921779 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:02.922696 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:34:02.922696 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:34:02.922696 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:02.922696 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:34:02.922696 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:02.922696 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:34:03.352486 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:34:03.353005 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:34:03.353136 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:34:03.358072 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:34:03.360599 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:03.366346 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:34:03.366346 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:03.366346 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:34:03.366346 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:34:03.366346 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:03.366346 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:34:03.366346 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:03.366346 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:34:03.367530 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.013 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:03.367530 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:34:03.367530 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:03.367530 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:34:03.367530 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:34:03.367530 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:34:03.367530 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:34:03.367530 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:03.367530 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:03.367530 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:03.367530 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:34:03.367530 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:03.367968 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:03.367968 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:03.367968 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:34:03.367968 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:03.367968 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:34:03.367968 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:34:03.367968 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:03.367968 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:34:03.367968 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:03.367968 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:34:03.367968 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:34:03.367968 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:03.367968 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:34:03.368379 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:03.368379 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:34:03.368724 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:03.368724 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:34:03.368724 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:34:03.368724 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:34:03.368724 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:34:03.368724 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:34:03.368724 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:03.368724 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:34:03.368724 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:34:03.368724 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:03.368724 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:34:03.368724 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:34:03.368724 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:03.368724 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:03.368724 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:34:03.368724 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:03.369188 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:34:03.369188 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:34:03.369188 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:03.369188 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:03.369188 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:03.369188 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:34:03.369188 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:03.369188 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:03.369188 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:03.369188 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:34:03.369188 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:03.369188 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:34:03.369188 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:34:03.369188 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:03.369188 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:34:03.369188 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:03.369188 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:34:03.369188 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:34:03.369188 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:03.369794 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:34:03.369794 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:03.369794 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:34:03.369794 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:34:03.369794 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:34:03.369794 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:34:03.370062 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:34:03.374305 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:34:03.374640 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:03.375342 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:34:03.375342 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:03.375342 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:34:03.375342 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:34:03.375342 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:03.375342 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:34:03.375342 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:03.375342 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:34:03.379583 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:03.379583 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:34:03.379583 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:34:03.379583 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:34:03.379583 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:03.379583 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:34:03.379583 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:03.379583 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:34:03.379583 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:03.379583 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:34:03.379583 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:03.379583 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:34:03.379583 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:03.379583 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:34:03.379583 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:34:03.380177 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:03.380177 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:34:03.380177 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:03.380177 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:34:03.380177 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:34:03.380177 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:03.380177 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:34:03.380177 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:03.380177 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:34:03.380177 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:03.380177 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:34:03.380177 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:03.380177 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:34:03.380177 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:34:03.380871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:34:03.380871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:34:03.380871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:34:03.380871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:34:03.380871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:34:03.380871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:03.380871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:03.380871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:03.380871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:34:03.380871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:03.380871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:03.380871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:03.380871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:34:03.381505 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:03.381505 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:34:03.381505 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:34:03.381505 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:03.381505 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:34:03.381505 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:03.381505 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:34:03.381505 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:34:03.381505 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:03.381505 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:34:03.381505 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:03.381505 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:34:03.381958 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:03.381958 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:34:03.381958 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:34:03.381958 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:34:03.381958 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:34:03.381958 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:34:03.381958 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:03.381958 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:34:03.381958 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:34:03.381958 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:03.381958 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:34:03.381958 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:34:03.381958 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:03.381958 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:03.381958 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:34:03.381958 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:03.382581 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:34:03.382581 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:34:03.382581 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:34:03.382581 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:34:03.382581 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:34:03.382581 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:03.382581 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:03.382581 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:03.382581 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:34:03.382581 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:03.382581 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:03.382581 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:03.382581 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:34:03.382581 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:03.382581 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:34:03.382581 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:34:03.382581 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:03.382581 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:34:03.383097 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:03.383097 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:34:03.383097 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:34:03.383097 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:03.383097 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:34:03.383097 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:03.383097 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:34:03.383097 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:34:03.383097 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:34:03.383097 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:34:03.383450 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:34:03.383450 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:34:03.383450 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:03.383450 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:34:03.383450 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:03.383450 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:34:03.383450 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:34:03.383450 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:03.383450 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:34:03.383450 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:03.383450 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:34:03.383450 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:03.383844 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:34:03.383844 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:34:03.383844 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:34:03.383844 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:03.383844 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:34:03.383844 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:03.383844 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:34:03.383844 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:03.383844 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:34:03.383844 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:03.383844 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:34:03.383844 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:03.383844 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:34:03.383844 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:34:03.383844 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:03.383844 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:34:03.383844 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:03.384392 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:34:03.384392 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:34:03.384392 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:03.384392 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:34:03.384392 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:03.384392 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:34:03.384392 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:03.384392 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:34:03.384392 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:03.384392 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:34:03.384392 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:34:03.384392 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:34:03.384392 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:34:03.384809 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:34:03.384809 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:34:03.384809 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:03.384809 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:03.384809 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:03.384809 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:34:03.384809 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:03.384809 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:03.384809 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:03.384809 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:34:03.384809 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:03.384809 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:34:03.384809 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:34:03.385222 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:03.385222 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:34:03.385222 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:03.385222 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:34:03.385222 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:34:03.385222 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:03.385222 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:34:03.385222 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:03.385222 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:34:03.386187 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:03.386187 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:34:03.386187 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:34:03.386187 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:34:03.386187 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:34:03.386187 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:34:03.386187 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:03.386187 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:34:03.386187 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:34:03.386187 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:03.386187 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:34:03.386187 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:34:03.386187 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:03.386187 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:03.386187 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:34:03.386187 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:03.386187 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:34:03.386739 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:34:03.386739 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:34:03.386739 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:34:03.386739 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:03.386739 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:03.386739 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:03.386739 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:34:03.386739 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:03.386739 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:03.386739 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:03.386739 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:34:03.386739 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:03.386739 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:34:03.386739 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:34:03.386739 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:03.386739 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:34:03.386739 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:03.386739 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:34:03.386739 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:34:03.387295 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:03.387295 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:34:03.387295 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:03.387295 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:34:03.387295 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:34:03.387295 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:34:03.387295 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:34:03.387295 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:34:03.387697 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:34:03.388165 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:34:03.388165 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:03.388789 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:34:03.388789 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:03.388789 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:34:03.388789 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:34:03.388789 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:03.388789 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:34:03.388789 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:03.388789 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:34:03.389422 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:03.389422 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:34:03.389422 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:03.389422 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:34:03.389422 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:34:03.389422 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:34:03.389422 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:34:03.389422 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:03.389422 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:03.389422 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:03.389422 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:34:03.389422 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:03.389809 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:03.389809 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:03.389809 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:34:03.389809 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:03.389809 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:34:03.389809 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:34:03.389809 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:03.389809 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:34:03.389809 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:03.389809 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:34:03.389809 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:34:03.389809 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:03.389809 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:34:03.390188 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:03.390188 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:34:03.390188 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:03.390188 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:34:03.390188 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:34:03.390188 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:34:03.390188 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:34:03.390188 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:34:03.390188 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:03.390188 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:34:03.390188 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:34:03.390188 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:03.390188 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:34:03.390188 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:34:03.390188 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:03.390645 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:03.390645 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:34:03.390645 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:03.390645 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:34:03.390645 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:34:03.390645 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:03.390645 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:03.390645 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:03.390645 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:34:03.390645 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:03.390645 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:03.390645 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:03.390645 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:34:03.390645 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:03.390645 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:34:03.390645 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:34:03.390645 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:03.390645 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:34:03.390645 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:03.391360 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:34:03.391360 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:34:03.391360 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:03.391360 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:34:03.391360 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:03.391360 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:34:07.885743 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:34:07.886896 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:34:07.887122 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:34:07.888654 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:34:07.921779 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:34:07.922161 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:07.923178 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:34:07.923178 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:07.923178 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:34:07.923178 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:34:07.923178 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:07.923178 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:34:07.923178 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:07.923178 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:34:07.924095 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.036 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:07.924095 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:34:07.924095 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:07.924095 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:34:07.924095 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:34:07.924095 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:34:07.924095 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:34:07.924095 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:07.924095 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:07.924095 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:07.924095 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:34:07.924095 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:07.924749 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:07.924749 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:07.924749 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:34:07.924749 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:07.924749 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:34:07.924749 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:34:07.924749 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:07.924749 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:34:07.924749 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:07.924749 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:34:07.924749 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:34:07.924749 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:07.924749 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:34:07.925304 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:07.925304 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:34:07.925304 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:07.925304 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:34:07.925304 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:34:07.925304 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:34:07.925304 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:34:07.925304 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:34:07.925304 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:07.925304 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:34:07.925304 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:34:07.925304 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:07.925304 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:34:07.925304 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:34:07.925304 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:07.926073 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:07.926073 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:34:07.926073 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:07.926073 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:34:07.926073 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:34:07.926073 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:07.926073 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:07.926073 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:07.926073 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:34:07.926073 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:07.926073 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:07.926073 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:07.926073 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:34:07.926073 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:07.926073 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:34:07.926073 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:34:07.926073 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:07.926073 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:34:07.926073 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:07.926970 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:34:07.926970 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:34:07.926970 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:07.926970 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:34:07.926970 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:07.926970 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:34:08.353214 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:34:08.354144 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:34:08.354374 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:34:08.355720 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:34:08.356128 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:08.357477 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:34:08.357477 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:08.357477 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:34:08.357477 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:34:08.357477 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:08.357477 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:34:08.357477 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:08.357477 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:34:08.359061 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:08.359061 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:34:08.359061 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:08.359061 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:34:08.359061 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:34:08.359061 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:34:08.359061 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:34:08.359061 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:08.359061 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:08.359061 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:08.359061 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:34:08.359061 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:08.359732 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:08.359732 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:08.359732 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:34:08.359732 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:08.359732 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:34:08.359732 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:34:08.359732 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:08.359732 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:34:08.359732 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:08.359732 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:34:08.359732 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:34:08.359732 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:08.359732 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:34:08.360300 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:08.360300 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:34:08.363259 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:08.363259 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:34:08.363259 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:34:08.363259 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:34:08.363259 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:34:08.363259 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:34:08.363259 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:08.363259 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:34:08.363259 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:34:08.363259 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:08.363259 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:34:08.363259 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:34:08.363259 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:08.363259 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:08.363259 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:34:08.363259 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:08.363727 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:34:08.363727 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:34:08.363727 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:08.363727 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:08.363727 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:08.363727 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:34:08.363727 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:08.363727 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:08.363727 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:08.363727 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:34:08.363727 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:08.363727 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:34:08.363727 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:34:08.363727 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:08.363727 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:34:08.363727 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:08.363727 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:34:08.363727 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:34:08.363727 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:08.364239 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:34:08.364239 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:08.364239 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:34:08.370042 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:34:08.370373 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:34:08.370482 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:34:08.370482 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:34:08.371258 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:34:08.371727 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:08.372578 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:34:08.372578 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:08.372578 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:34:08.372578 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:34:08.372578 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:08.372578 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:34:08.372578 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:08.372578 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:34:08.373158 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:08.373158 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:34:08.373158 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:34:08.373158 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:34:08.373158 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:08.373158 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:34:08.373158 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:08.373158 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:34:08.373158 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:08.373158 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:34:08.373158 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:08.373158 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:34:08.373158 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:08.373158 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:34:08.373158 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:34:08.374155 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:08.374155 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:34:08.374155 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:08.374155 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:34:08.374155 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:34:08.374155 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:08.374155 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:34:08.374155 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:08.374155 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:34:08.374155 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:08.374155 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:34:08.374155 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:08.374155 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:34:08.374155 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:34:08.375082 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:34:08.375082 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:34:08.375082 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:34:08.375082 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:34:08.375082 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:34:08.375082 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:08.375082 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:08.375082 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:08.375082 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:34:08.375082 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:08.375082 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:08.375082 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:08.375082 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:34:08.375935 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:08.375935 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:34:08.375935 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:34:08.375935 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:08.375935 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:34:08.375935 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:08.375935 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:34:08.375935 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:34:08.375935 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:08.375935 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:34:08.375935 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:08.375935 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:34:08.376705 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:08.376705 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:34:08.376705 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:34:08.376705 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:34:08.376705 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:34:08.376705 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:34:08.376705 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:08.376705 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:34:08.376705 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:34:08.376705 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:08.376705 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:34:08.376705 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:34:08.376705 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:08.376705 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:08.376705 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:34:08.376705 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:08.377695 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:34:08.377695 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:34:08.377695 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:34:08.377695 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:34:08.377695 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:34:08.377695 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:08.377695 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:08.377695 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:08.377695 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:34:08.377695 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:08.377695 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:08.377695 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:08.377695 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:34:08.377695 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:08.377695 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:34:08.377695 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:34:08.377695 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:08.377695 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:34:08.378779 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:08.378779 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:34:08.378779 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:34:08.378779 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:08.378779 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:34:08.378779 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:08.378779 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:34:08.380289 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:34:08.380868 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:34:08.380868 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:34:08.380868 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:34:08.381276 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:34:08.381681 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:08.385345 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:34:08.385345 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:08.385345 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:34:08.385345 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:34:08.385345 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:08.385345 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:34:08.385345 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:08.385345 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:34:08.385345 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:08.385345 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:34:08.385345 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:34:08.385345 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:34:08.385345 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:08.385345 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:34:08.385345 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:08.385345 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:34:08.385345 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:08.385345 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:34:08.385866 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:08.385866 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:34:08.385866 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:08.385866 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:34:08.385866 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:34:08.385866 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:08.385866 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:34:08.385866 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:08.385866 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:34:08.385866 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:34:08.385866 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:08.385866 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:34:08.385866 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:08.385866 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:34:08.385866 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:08.386264 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:34:08.386264 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:08.386264 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:34:08.386264 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:34:08.386264 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:34:08.386264 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:34:08.386264 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:34:08.386264 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:34:08.386264 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:08.386264 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:08.386264 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:08.386264 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:34:08.386264 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:08.386624 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:08.386624 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:08.386624 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:34:08.386624 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:08.386624 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:34:08.386624 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:34:08.386624 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:08.386624 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:34:08.386624 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:08.386624 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:34:08.386624 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:34:08.386624 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:08.386624 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:34:08.386926 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:08.386926 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:34:08.386926 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:08.386926 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:34:08.386926 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:34:08.386926 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:34:08.386926 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:34:08.386926 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:34:08.386926 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:08.386926 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:34:08.386926 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:34:08.386926 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:08.386926 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:34:08.386926 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:34:08.386926 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:08.387332 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:08.387332 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:34:08.387332 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:08.387332 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:34:08.387332 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:34:08.387332 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:34:08.387332 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:34:08.387332 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:08.387332 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:08.387332 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:08.387332 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:34:08.387332 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:08.387332 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:08.387332 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:08.387332 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:34:08.387332 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:08.387332 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:34:08.387332 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:34:08.387332 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:08.387741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:34:08.387741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:08.387741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:34:08.387741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:34:08.387741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:08.387741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:34:08.387741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:08.387741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:34:08.387741 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:34:08.389603 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:34:08.389717 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:34:08.389949 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:34:08.392968 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:34:08.394071 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:34:08.394783 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:08.396299 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:34:08.396299 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:08.396299 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:34:08.396299 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:34:08.396299 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:08.396299 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:34:08.396299 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:08.396299 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:34:08.396868 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:08.396868 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:34:08.396868 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:08.396868 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:34:08.396868 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:34:08.396868 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:34:08.396868 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:34:08.396868 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:08.396868 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:08.396868 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:08.396868 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:34:08.396868 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:08.397235 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:08.397235 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:08.397235 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:34:08.397235 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:08.397235 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:34:08.397235 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:34:08.397235 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:08.397235 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:34:08.397235 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:08.397235 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:34:08.397235 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:34:08.397235 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:08.397235 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:34:08.397638 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:08.397638 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:34:08.397959 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:08.397959 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:34:08.397959 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:34:08.397959 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:34:08.397959 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:34:08.397959 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:34:08.397959 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:08.397959 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:34:08.397959 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:34:08.397959 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:08.397959 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:34:08.397959 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:34:08.397959 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:08.397959 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:08.397959 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:34:08.397959 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:08.398464 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:34:08.398464 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:34:08.398464 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:08.398464 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:08.398464 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:08.398464 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:34:08.398464 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:08.398464 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:08.398464 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:08.398464 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:34:08.398464 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:08.398464 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:34:08.398464 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:34:08.398464 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:08.398464 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:34:08.398464 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:08.398464 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:34:08.398464 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:34:08.398464 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:08.399021 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:34:08.399021 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:08.399021 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:34:12.886594 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:34:12.890212 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:34:12.890212 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:34:12.890212 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:34:12.921505 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:34:12.921832 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:12.924341 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:34:12.924341 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:12.924341 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:34:12.924341 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:34:12.924341 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:12.924341 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:34:12.924341 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:12.924341 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:34:12.924341 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.035 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:12.924341 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:34:12.924341 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:12.924341 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:34:12.924341 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:34:12.924341 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:34:12.924341 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:34:12.924698 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:12.924698 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:12.924698 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:12.924698 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:34:12.924698 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:12.924698 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:12.924698 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:12.924698 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:34:12.924698 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:12.924698 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:34:12.924698 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:34:12.924698 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:12.924698 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:34:12.924698 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:12.925052 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:34:12.925052 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:34:12.925052 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:12.925052 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:34:12.925052 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:12.925052 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:34:12.925052 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:12.925052 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:34:12.925052 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:34:12.925052 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:34:12.925052 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:34:12.925052 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:34:12.925052 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:12.925052 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:34:12.925339 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:34:12.925339 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:12.925339 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:34:12.925339 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:34:12.925339 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:12.925339 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:12.925339 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:34:12.925339 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:12.925339 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:34:12.925339 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:34:12.925339 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:12.925339 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:12.925339 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:12.925339 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:34:12.925339 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:12.925339 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:12.925339 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:12.925339 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:34:12.925339 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:12.925339 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:34:12.925339 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:34:12.925788 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:12.925788 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:34:12.925788 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:12.925788 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:34:12.925788 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:34:12.925788 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:12.925788 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:34:12.925788 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:12.925788 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:34:13.354090 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:34:13.357292 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:34:13.357292 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:34:13.357974 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:34:13.358380 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:13.369529 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:34:13.369529 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:13.369529 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:34:13.369529 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:34:13.369529 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:13.369529 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:34:13.369529 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:13.369529 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:34:13.369529 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.011 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:13.369529 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:34:13.369529 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:13.369529 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:34:13.369529 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:34:13.369529 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:34:13.369529 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:34:13.369911 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:13.369911 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:13.369911 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:13.369911 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:34:13.369911 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:13.369911 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:13.369911 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:13.369911 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:34:13.369911 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:13.369911 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:34:13.369911 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:34:13.369911 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:13.369911 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:34:13.369911 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:13.370261 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:34:13.370261 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:34:13.370261 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:13.370261 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:34:13.370261 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:13.370261 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:34:13.370261 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:13.370261 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:34:13.370261 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:34:13.370261 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:34:13.370261 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:34:13.370261 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:34:13.370261 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:13.370261 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:34:13.370621 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:34:13.370621 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:13.370621 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:34:13.370621 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:34:13.370621 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:13.370621 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:13.370621 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:34:13.370621 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:13.370621 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:34:13.370621 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:34:13.370621 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:13.370621 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:13.370621 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:13.370621 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:34:13.370621 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:13.370621 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:13.370621 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:13.370621 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:34:13.370621 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:13.370621 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:34:13.371062 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:34:13.371062 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:13.371062 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:34:13.371062 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:13.371062 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:34:13.371062 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:34:13.371062 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:13.371062 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:34:13.371062 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:13.371062 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:34:13.371062 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:34:13.376873 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:34:13.376999 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:34:13.377123 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:34:13.377980 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:34:13.378313 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:13.379614 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:34:13.379614 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:13.379614 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:34:13.379614 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:34:13.379614 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:13.379614 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:34:13.379614 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:13.379614 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:34:13.381213 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:13.381213 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:34:13.381213 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:34:13.381213 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:34:13.381213 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:13.381213 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:34:13.381213 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:13.381213 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:34:13.381213 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:13.381213 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:34:13.381213 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:13.381213 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:34:13.381213 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:13.381213 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:34:13.381213 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:34:13.381549 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:13.381549 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:34:13.381549 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:13.381549 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:34:13.381549 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:34:13.381549 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:13.381549 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:34:13.381549 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:13.381549 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:34:13.382357 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:13.382357 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:34:13.382357 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:13.382357 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:34:13.382357 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:34:13.382357 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:34:13.382357 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:34:13.382357 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:34:13.382357 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:34:13.382357 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:34:13.382357 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:13.382357 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:13.382619 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:13.382619 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:34:13.382619 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:13.382619 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:13.382619 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:13.382619 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:34:13.382619 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:13.382619 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:34:13.382619 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:34:13.382619 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:13.382619 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:34:13.382619 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:13.382619 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:34:13.382619 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:34:13.382908 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:13.382908 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:34:13.382908 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:13.382908 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:34:13.383770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:13.383770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:34:13.383770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:34:13.383770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:34:13.383770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:34:13.383770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:34:13.383770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:13.383770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:34:13.383770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:34:13.383770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:13.383770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:34:13.383770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:34:13.383770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:13.383770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:13.383770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:34:13.383770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:13.384083 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:34:13.384083 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:34:13.384083 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:34:13.384083 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:34:13.384083 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:34:13.384083 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:13.384083 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:13.384083 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:13.384083 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:34:13.384083 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:13.384083 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:13.384083 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:13.384083 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:34:13.384083 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:13.384083 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:34:13.384083 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:34:13.384083 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:13.384083 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:34:13.384457 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:13.384457 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:34:13.384457 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:34:13.384457 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:13.384457 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:34:13.384457 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:13.384457 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:34:13.384457 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:34:13.384457 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:34:13.384672 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:34:13.384729 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:34:13.385678 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:34:13.386075 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:13.386944 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:34:13.386944 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:13.386944 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:34:13.386944 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:34:13.386944 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:13.386944 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:34:13.386944 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:13.386944 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:34:13.387801 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:13.387801 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:34:13.387801 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:34:13.387801 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:34:13.387801 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:13.387801 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:34:13.387801 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:13.387801 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:34:13.387801 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:13.387801 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:34:13.387801 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:13.387801 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:34:13.387801 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:13.387801 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:34:13.387801 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:34:13.388121 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:13.388121 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:34:13.388121 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:13.388121 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:34:13.388121 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:34:13.388121 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:13.388121 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:34:13.388121 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:13.388121 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:34:13.388871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:13.388871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:34:13.388871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:13.388871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:34:13.388871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:34:13.388871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:34:13.388871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:34:13.388871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:34:13.388871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:34:13.388871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:13.388871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:13.389133 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:13.389133 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:34:13.389133 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:13.389133 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:13.389133 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:13.389133 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:34:13.389133 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:13.389133 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:34:13.389133 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:34:13.389133 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:13.389133 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:34:13.389133 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:13.389133 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:34:13.389133 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:34:13.389455 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:13.389455 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:34:13.389455 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:13.389455 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:34:13.390072 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:13.390072 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:34:13.390072 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:34:13.390072 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:34:13.390072 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:34:13.390072 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:34:13.390072 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:13.390072 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:34:13.390072 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:34:13.390072 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:13.390072 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:34:13.390072 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:34:13.390072 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:13.390072 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:13.390072 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:34:13.390072 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:13.390072 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:34:13.390455 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:34:13.390455 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:34:13.390455 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:34:13.390455 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:13.390455 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:13.390455 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:13.390455 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:34:13.390455 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:13.390455 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:13.390455 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:13.390455 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:34:13.390455 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:13.390455 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:34:13.390455 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:34:13.390455 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:13.390455 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:34:13.390455 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:13.390455 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:34:13.390455 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:34:13.390848 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:13.390848 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:34:13.390848 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:13.390848 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:34:13.390848 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:34:13.390848 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:34:13.391026 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:34:13.391081 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:34:13.395202 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:34:13.395202 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:34:13.395202 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:13.395202 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:34:13.395202 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:13.395202 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:34:13.395202 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:34:13.395202 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:13.395202 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:34:13.395202 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:13.395202 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:34:13.395202 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:13.395741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:34:13.395741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:13.395741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:34:13.395741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:34:13.395741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:34:13.395741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:34:13.395741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:13.395741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:13.395741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:13.395741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:34:13.395741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:13.395741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:13.395741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:13.395741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:34:13.396053 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:13.396053 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:34:13.396053 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:34:13.396053 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:13.396053 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:34:13.396053 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:13.396053 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:34:13.396053 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:34:13.396053 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:13.396053 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:34:13.396053 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:13.396053 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:34:13.396345 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:13.396345 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:34:13.396345 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:34:13.396345 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:34:13.396345 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:34:13.396345 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:34:13.396345 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:13.396345 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:34:13.396345 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:34:13.396345 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:13.396345 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:34:13.396345 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:34:13.396345 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:13.396345 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:13.396345 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:34:13.396345 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:13.396665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:34:13.396665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:34:13.396665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:13.396665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:13.396665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:13.396665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:34:13.396665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:13.396665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:13.396665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:13.396665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:34:13.396665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:13.396665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:34:13.396665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:34:13.396665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:13.396665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:34:13.396665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:13.396665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:34:13.396665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:34:13.396665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:13.397067 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:34:13.397067 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:13.397067 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:34:17.888855 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:34:17.889763 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:34:17.889763 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:34:17.891177 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:34:17.925505 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:34:17.926026 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:17.927026 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:34:17.927026 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:17.927026 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:34:17.927026 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:34:17.927026 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:17.927026 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:34:17.927026 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:17.927026 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:34:17.927853 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.037 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:17.927853 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:34:17.927853 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:17.927853 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:34:17.927853 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:34:17.927853 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:34:17.927853 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:34:17.927853 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:17.927853 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:17.927853 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:17.927853 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:34:17.927853 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:17.928499 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:17.928499 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:17.928499 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:34:17.928499 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:17.928499 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:34:17.928499 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:34:17.928499 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:17.928499 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:34:17.928499 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:17.928499 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:34:17.928499 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:34:17.928499 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:17.928499 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:34:17.929100 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:17.929100 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:34:17.929100 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:17.929100 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:34:17.929100 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:34:17.929100 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:34:17.929100 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:34:17.929100 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:34:17.929100 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:17.929100 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:34:17.929100 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:34:17.929100 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:17.929100 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:34:17.929100 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:34:17.929100 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:17.929858 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:17.929858 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:34:17.929858 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:17.929858 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:34:17.929858 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:34:17.929858 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:17.929858 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:17.929858 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:17.929858 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:34:17.929858 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:17.929858 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:17.929858 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:17.929858 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:34:17.929858 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:17.929858 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:34:17.929858 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:34:17.929858 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:17.929858 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:34:17.929858 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:17.930805 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:34:17.930805 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:34:17.930805 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:17.930805 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:34:17.930805 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:17.930805 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:34:18.357138 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:34:18.357604 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:34:18.357791 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:34:18.359138 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:34:18.359574 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:18.360580 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:34:18.360580 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:18.360580 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:34:18.360580 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:34:18.360580 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:18.360580 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:34:18.360580 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:18.360580 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:34:18.362192 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:18.362192 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:34:18.362192 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:18.362192 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:34:18.362192 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:34:18.362192 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:34:18.362192 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:34:18.362192 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:18.362192 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:18.362192 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:18.362192 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:34:18.362192 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:18.362899 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:18.362899 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:18.362899 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:34:18.362899 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:18.362899 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:34:18.362899 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:34:18.362899 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:18.362899 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:34:18.362899 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:18.362899 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:34:18.362899 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:34:18.362899 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:18.362899 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:34:18.363525 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:18.363525 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:34:18.363847 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:18.363847 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:34:18.363847 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:34:18.363847 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:34:18.363847 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:34:18.363847 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:34:18.363847 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:18.363847 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:34:18.363847 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:34:18.363847 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:18.363847 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:34:18.363847 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:34:18.363847 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:18.363847 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:18.363847 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:34:18.363847 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:18.364617 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:34:18.364617 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:34:18.364617 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:18.364617 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:18.364617 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:18.364617 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:34:18.364617 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:18.364617 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:18.364617 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:18.364617 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:34:18.364617 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:18.364617 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:34:18.364617 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:34:18.364617 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:18.364617 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:34:18.364617 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:18.364617 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:34:18.364617 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:34:18.364617 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:18.365520 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:34:18.365520 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:18.365520 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:34:18.377153 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:34:18.377524 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:34:18.377712 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:34:18.377858 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:34:18.378448 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:34:18.378738 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:18.379608 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:34:18.379608 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:18.379608 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:34:18.379608 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:34:18.379608 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:18.379608 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:34:18.379608 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:18.379608 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:34:18.380807 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:18.380807 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:34:18.380807 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:34:18.380807 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:34:18.380807 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:18.380807 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:34:18.380807 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:18.380807 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:34:18.380807 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:18.380807 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:34:18.380807 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:18.380807 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:34:18.380807 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:18.380807 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:34:18.380807 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:34:18.381215 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:18.381215 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:34:18.381215 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:18.381215 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:34:18.381215 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:34:18.381215 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:18.381215 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:34:18.381215 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:18.381215 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:34:18.382185 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:18.382185 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:34:18.382185 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:18.382185 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:34:18.382185 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:34:18.382185 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:34:18.382185 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:34:18.382185 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:34:18.382185 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:34:18.382185 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:34:18.382185 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:18.382185 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:18.382553 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:18.382553 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:34:18.382553 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:18.382553 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:18.382553 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:18.382553 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:34:18.382553 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:18.382553 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:34:18.382553 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:34:18.382553 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:18.382553 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:34:18.382553 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:18.382553 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:34:18.382553 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:34:18.382907 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:18.382907 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:34:18.382907 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:18.382907 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:34:18.384088 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:18.384088 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:34:18.384088 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:34:18.384088 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:34:18.384088 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:34:18.384088 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:34:18.384088 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:18.384088 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:34:18.384088 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:34:18.384088 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:18.384088 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:34:18.384088 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:34:18.384088 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:18.384088 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:18.384088 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:34:18.384088 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:18.384538 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:34:18.384538 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:34:18.384538 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:34:18.384538 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:34:18.384538 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:34:18.384538 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:18.384538 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:18.384538 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:18.384538 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:34:18.384538 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:18.384538 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:18.384538 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:18.384538 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:34:18.384538 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:18.384538 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:34:18.384538 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:34:18.384538 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:18.384538 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:34:18.385204 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:18.385204 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:34:18.385204 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:34:18.385204 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:18.385204 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:34:18.385204 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:18.385204 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:34:18.385204 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:34:18.385204 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:34:18.385204 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:34:18.385516 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:34:18.385516 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:34:18.385703 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:18.386506 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:34:18.386506 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:18.386506 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:34:18.386506 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:34:18.386506 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:18.386506 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:34:18.386506 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:18.386506 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:34:18.387334 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:18.387334 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:34:18.387334 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:34:18.387334 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:34:18.387334 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:18.387334 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:34:18.387334 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:18.387334 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:34:18.387334 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:18.387334 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:34:18.387334 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:18.387334 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:34:18.387334 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:18.387334 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:34:18.387334 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:34:18.387734 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:18.387734 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:34:18.387734 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:18.387734 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:34:18.387734 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:34:18.387734 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:18.387734 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:34:18.387734 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:18.387734 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:34:18.388236 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:18.388236 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:34:18.388236 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:18.388236 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:34:18.388236 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:34:18.388236 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:34:18.388236 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:34:18.388236 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:34:18.388236 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:34:18.388236 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:18.388236 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:18.388559 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:18.388559 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:34:18.388559 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:18.388559 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:18.388559 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:18.388559 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:34:18.388559 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:18.388559 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:34:18.388559 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:34:18.388559 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:18.388559 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:34:18.388559 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:18.388559 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:34:18.388559 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:34:18.388915 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:18.388915 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:34:18.388915 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:18.388915 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:34:18.389482 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:18.389482 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:34:18.389482 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:34:18.389482 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:34:18.389482 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:34:18.389482 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:34:18.389482 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:18.389482 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:34:18.389482 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:34:18.389482 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:18.389482 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:34:18.389482 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:34:18.389482 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:18.389482 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:18.389482 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:34:18.389482 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:18.389482 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:34:18.389962 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:34:18.389962 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:34:18.389962 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:34:18.389962 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:18.389962 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:18.389962 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:18.389962 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:34:18.389962 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:18.389962 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:18.389962 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:18.389962 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:34:18.389962 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:18.389962 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:34:18.389962 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:34:18.389962 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:18.389962 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:34:18.389962 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:18.389962 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:34:18.389962 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:34:18.390459 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:18.390459 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:34:18.390459 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:18.390459 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:34:18.390768 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:34:18.390990 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:34:18.391101 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:34:18.391295 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:34:18.391631 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:34:18.391849 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:34:18.391970 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:18.392541 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:34:18.392541 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:18.392541 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:34:18.392541 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:34:18.392541 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:18.392541 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:34:18.392541 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:18.392541 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:34:18.393424 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:18.393424 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:34:18.393424 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:18.393424 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:34:18.393424 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:34:18.393424 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:34:18.393424 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:34:18.393424 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:18.393424 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:18.393424 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:18.393424 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:34:18.393424 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:18.393766 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:18.393766 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:18.393766 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:34:18.393766 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:18.393766 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:34:18.393766 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:34:18.393766 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:18.393766 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:34:18.393766 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:18.393766 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:34:18.393766 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:34:18.393766 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:18.393766 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:34:18.394093 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:18.394093 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:34:18.394490 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:18.394490 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:34:18.394490 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:34:18.394490 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:34:18.394490 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:34:18.394490 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:34:18.394490 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:18.394490 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:34:18.394490 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:34:18.394490 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:18.394490 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:34:18.394490 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:34:18.394490 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:18.394490 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:18.394490 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:34:18.394490 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:18.394915 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:34:18.394915 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:34:18.394915 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:18.394915 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:18.394915 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:18.394915 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:34:18.394915 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:18.394915 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:18.394915 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:18.394915 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:34:18.394915 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:18.394915 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:34:18.394915 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:34:18.394915 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:18.394915 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:34:18.394915 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:18.394915 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:34:18.394915 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:34:18.394915 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:18.395443 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:34:18.395443 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:18.395443 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:34:22.889741 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:34:22.893039 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:34:22.893039 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:34:22.893039 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:34:22.924287 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:34:22.924675 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:22.925431 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:34:22.925431 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:22.925431 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:34:22.925431 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:34:22.925431 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:22.925431 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:34:22.925431 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:22.925431 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:34:22.926906 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.035 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:22.926906 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:34:22.926906 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:22.926906 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:34:22.926906 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:34:22.926906 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:34:22.926906 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:34:22.926906 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:22.926906 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:22.926906 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:22.926906 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:34:22.926906 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:22.927232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:22.927232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:22.927232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:34:22.927232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:22.927232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:34:22.927232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:34:22.927232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:22.927232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:34:22.927232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:22.927232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:34:22.927232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:34:22.927232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:22.927232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:34:22.927500 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:22.927500 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:34:22.927821 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:22.927821 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:34:22.927821 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:34:22.927821 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:34:22.927821 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:34:22.927821 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:34:22.927821 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:22.927821 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:34:22.927821 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:34:22.927821 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:22.927821 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:34:22.927821 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:34:22.927821 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:22.927821 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:22.927821 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:34:22.927821 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:22.928449 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:34:22.928449 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:34:22.928449 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:22.928449 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:22.928449 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:22.928449 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:34:22.928449 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:22.928449 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:22.928449 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:22.928449 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:34:22.928449 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:22.928449 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:34:22.928449 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:34:22.928449 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:22.928449 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:34:22.928449 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:22.928449 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:34:22.928449 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:34:22.928449 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:22.929007 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:34:22.929007 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:22.929007 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:34:23.357915 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:34:23.359364 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:34:23.359364 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:34:23.360074 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:34:23.360477 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:23.364368 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:34:23.364368 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:23.364368 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:34:23.364368 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:34:23.364368 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:23.364368 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:34:23.364368 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:23.364368 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:34:23.364368 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:23.364368 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:34:23.364368 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:23.364368 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:34:23.364368 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:34:23.364368 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:34:23.364368 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:34:23.365105 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:23.365105 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:23.365105 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:23.365105 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:34:23.365105 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:23.365105 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:23.365105 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:23.365105 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:34:23.365105 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:23.365105 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:34:23.365105 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:34:23.365105 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:23.365105 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:34:23.365105 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:23.365729 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:34:23.365729 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:34:23.365729 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:23.365729 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:34:23.365729 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:23.365729 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:34:23.366816 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:23.366816 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:34:23.366816 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:34:23.366816 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:34:23.366816 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:34:23.366816 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:34:23.366816 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:23.366816 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:34:23.366816 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:34:23.366816 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:23.366816 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:34:23.366816 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:34:23.366816 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:23.366816 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:23.366816 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:34:23.366816 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:23.367405 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:34:23.367405 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:34:23.367405 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:23.367405 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:23.367405 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:23.367405 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:34:23.367405 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:23.367405 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:23.367405 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:23.367405 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:34:23.367405 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:23.367405 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:34:23.367405 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:34:23.367405 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:23.367405 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:34:23.367405 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:23.367405 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:34:23.367405 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:34:23.367405 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:23.367894 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:34:23.367894 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:23.367894 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:34:23.377630 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:34:23.377950 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:34:23.378157 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:34:23.378466 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:34:23.379279 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:34:23.379682 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:23.380639 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:34:23.380639 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:23.380639 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:34:23.380639 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:34:23.380639 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:23.380639 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:34:23.380639 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:23.380639 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:34:23.384222 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:23.384222 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:34:23.384222 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:34:23.384222 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:34:23.384222 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:23.384222 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:34:23.384222 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:23.384222 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:34:23.384222 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:23.384222 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:34:23.384222 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:23.384222 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:34:23.384222 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:23.384222 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:34:23.384222 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:34:23.385153 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:23.385153 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:34:23.385153 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:23.385153 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:34:23.385153 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:34:23.385153 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:23.385153 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:34:23.385153 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:23.385153 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:34:23.385153 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:23.385153 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:34:23.385153 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:23.385153 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:34:23.385153 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:34:23.385844 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:34:23.385844 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:34:23.385844 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:34:23.385844 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:34:23.385844 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:34:23.385844 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:23.385844 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:23.385844 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:23.385844 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:34:23.385844 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:23.385844 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:23.385844 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:23.385844 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:34:23.386507 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:23.386507 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:34:23.386507 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:34:23.386507 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:23.386507 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:34:23.386507 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:23.386507 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:34:23.386507 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:34:23.386507 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:23.386507 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:34:23.386507 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:23.386507 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:34:23.387604 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:23.387604 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:34:23.387604 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:34:23.387604 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:34:23.387604 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:34:23.387604 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:34:23.387604 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:23.387604 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:34:23.387604 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:34:23.387604 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:23.387604 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:34:23.387604 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:34:23.387604 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:23.387604 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:23.387604 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:34:23.387604 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:23.387964 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:34:23.387964 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:34:23.387964 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:34:23.387964 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:34:23.387964 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:34:23.387964 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:23.387964 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:23.387964 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:23.387964 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:34:23.387964 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:23.387964 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:23.387964 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:23.387964 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:34:23.387964 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:23.387964 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:34:23.387964 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:34:23.387964 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:23.387964 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:34:23.388289 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:23.388289 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:34:23.388289 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:34:23.388289 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:23.388289 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:34:23.388289 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:23.388289 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:34:23.388584 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:34:23.388826 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:34:23.388974 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:34:23.389131 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:34:23.389704 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:34:23.390058 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:23.390915 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:34:23.390915 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:23.390915 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:34:23.390915 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:34:23.390915 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:23.390915 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:34:23.390915 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:23.390915 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:34:23.391856 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:23.391856 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:34:23.391856 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:34:23.391856 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:34:23.391856 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:23.391856 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:34:23.391856 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:23.391856 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:34:23.391856 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:23.391856 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:34:23.391856 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:23.391856 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:34:23.391856 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:23.391856 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:34:23.391856 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:34:23.392158 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:23.392158 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:34:23.392158 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:23.392158 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:34:23.392158 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:34:23.392158 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:23.392158 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:34:23.392158 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:23.392158 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:34:23.392461 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:34:23.393136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:23.393136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:34:23.393136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:23.393136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:34:23.393136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:34:23.393136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:34:23.393136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:34:23.393136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:34:23.393136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:34:23.393136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:23.393136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:23.393402 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:23.393402 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:34:23.393402 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:23.393402 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:23.393402 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:23.393402 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:34:23.393402 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:23.393402 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:34:23.393402 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:34:23.393402 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:23.393402 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:34:23.393402 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:23.393402 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:34:23.393402 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:34:23.393667 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:23.393667 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:34:23.393667 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:23.393667 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:34:23.394699 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:23.394699 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:34:23.394699 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:34:23.394699 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:34:23.394699 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:34:23.394699 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:34:23.394699 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:23.394699 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:34:23.394699 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:34:23.394699 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:23.394699 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:34:23.394699 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:34:23.394699 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:23.394699 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:23.394699 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:34:23.394699 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:23.394699 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:34:23.395070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:34:23.395070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:34:23.395070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:34:23.395070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:23.395070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:23.395070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:23.395070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:34:23.395070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:23.395070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:23.395070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:23.395070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:34:23.395070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:23.395070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:34:23.395070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:34:23.395070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:23.395070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:34:23.395070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:23.395070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:34:23.395070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:34:23.395422 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:23.395422 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:34:23.395422 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:23.395422 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:34:23.395787 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:34:23.395787 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:34:23.395787 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:34:23.396205 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:34:23.396708 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:34:23.396857 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:23.398938 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:34:23.398938 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:23.398938 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:34:23.398938 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:34:23.398938 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:23.398938 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:34:23.398938 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:23.398938 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:34:23.398938 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:23.398938 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:34:23.398938 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:23.398938 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:34:23.398938 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:34:23.398938 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:34:23.398938 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:34:23.399256 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:23.399256 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:23.399256 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:23.399256 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:34:23.399256 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:23.399256 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:23.399256 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:23.399256 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:34:23.399256 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:23.399256 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:34:23.399256 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:34:23.399256 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:23.399256 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:34:23.399256 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:23.399523 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:34:23.399523 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:34:23.399523 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:23.399523 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:34:23.399523 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:23.399523 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:34:23.399523 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:23.399523 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:34:23.399523 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:34:23.399523 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:34:23.399523 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:34:23.399523 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:34:23.399523 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:23.399523 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:34:23.399777 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:34:23.399777 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:23.399777 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:34:23.399777 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:34:23.399777 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:23.399777 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:23.399777 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:34:23.399777 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:23.399777 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:34:23.399777 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:34:23.399777 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:23.399777 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:23.399777 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:23.399777 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:34:23.399777 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:23.399777 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:23.399777 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:23.399777 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:34:23.399777 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:23.399777 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:34:23.400123 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:34:23.400123 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:23.400123 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:34:23.400123 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:23.400123 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:34:23.400123 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:34:23.400123 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:23.400123 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:34:23.400123 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:23.400123 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:34:27.891057 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:34:27.894392 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:34:27.894392 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:34:27.894392 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:34:27.932419 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:34:27.932835 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:27.936540 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:34:27.936540 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:27.936540 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:34:27.936540 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:34:27.936540 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:27.936540 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:34:27.936540 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:27.936540 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:34:27.936540 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.041 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:27.936540 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:34:27.936540 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:27.936540 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:34:27.936540 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:34:27.936540 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:34:27.936540 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:34:27.936985 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:27.936985 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:27.936985 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:27.936985 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:34:27.936985 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:27.936985 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:27.936985 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:27.936985 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:34:27.936985 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:27.936985 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:34:27.936985 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:34:27.936985 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:27.936985 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:34:27.936985 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:27.937417 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:34:27.937417 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:34:27.937417 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:27.937417 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:34:27.937417 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:27.937417 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:34:27.937417 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:27.937417 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:34:27.937417 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:34:27.937417 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:34:27.937417 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:34:27.937417 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:34:27.937417 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:27.937417 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:34:27.938390 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:34:27.938390 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:27.938390 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:34:27.938390 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:34:27.938390 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:27.938390 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:27.938390 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:34:27.938390 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:27.938390 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:34:27.938390 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:34:27.938390 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:27.938390 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:27.938390 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:27.938390 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:34:27.938390 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:27.938390 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:27.938390 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:27.938390 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:34:27.938390 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:27.938390 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:34:27.938390 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:34:27.939008 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:27.939008 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:34:27.939008 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:27.939008 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:34:27.939008 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:34:27.939008 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:27.939008 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:34:27.939008 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:27.939008 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:34:28.358959 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:34:28.359572 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:34:28.359811 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:34:28.360843 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:34:28.361136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:28.361998 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:34:28.361998 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:28.361998 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:34:28.361998 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:34:28.361998 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:28.361998 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:34:28.361998 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:28.361998 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:34:28.363910 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:28.363910 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:34:28.363910 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:28.363910 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:34:28.363910 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:34:28.363910 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:34:28.363910 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:34:28.363910 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:28.363910 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:28.363910 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:28.363910 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:34:28.363910 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:28.364352 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:28.364352 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:28.364352 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:34:28.364352 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:28.364352 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:34:28.364352 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:34:28.364352 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:28.364352 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:34:28.364352 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:28.364352 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:34:28.364352 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:34:28.364352 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:28.364352 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:34:28.364799 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:28.364799 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:34:28.364799 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:28.364799 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:34:28.364799 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:34:28.364799 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:34:28.364799 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:34:28.364799 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:34:28.364799 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:28.364799 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:34:28.364799 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:34:28.364799 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:28.364799 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:34:28.364799 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:34:28.364799 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:28.365273 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:28.365273 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:34:28.365273 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:28.365273 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:34:28.365273 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:34:28.365273 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:28.365273 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:28.365273 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:28.365273 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:34:28.365273 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:28.365273 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:28.365273 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:28.365273 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:34:28.365273 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:28.365273 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:34:28.365273 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:34:28.365273 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:28.365273 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:34:28.365273 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:28.365809 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:34:28.365809 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:34:28.365809 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:28.365809 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:34:28.365809 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:28.365809 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:34:28.378569 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:34:28.378569 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:34:28.378569 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:34:28.378569 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:34:28.378831 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:34:28.379135 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:28.379997 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:34:28.379997 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:28.379997 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:34:28.379997 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:34:28.379997 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:28.379997 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:34:28.379997 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:28.379997 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:34:28.381388 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:28.381388 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:34:28.381388 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:34:28.381388 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:34:28.381388 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:28.381388 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:34:28.381388 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:28.381388 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:34:28.381388 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:28.381388 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:34:28.381388 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:28.381388 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:34:28.381388 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:28.381388 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:34:28.381388 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:34:28.383149 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:28.383149 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:34:28.383149 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:28.383149 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:34:28.383149 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:34:28.383149 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:28.383149 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:34:28.383149 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:28.383149 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:34:28.383149 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:28.383149 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:34:28.383149 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:28.383149 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:34:28.383149 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:34:28.383638 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:34:28.383638 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:34:28.383638 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:34:28.383638 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:34:28.383638 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:34:28.383638 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:28.383638 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:28.383638 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:28.383638 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:34:28.383638 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:28.383638 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:28.383638 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:28.383638 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:34:28.384055 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:28.384055 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:34:28.384055 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:34:28.384055 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:28.384055 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:34:28.384055 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:28.384055 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:34:28.384055 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:34:28.384055 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:28.384055 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:34:28.384055 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:28.384055 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:34:28.384435 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:28.384435 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:34:28.384435 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:34:28.384435 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:34:28.384435 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:34:28.384435 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:34:28.384435 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:28.384435 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:34:28.384435 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:34:28.384435 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:28.384435 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:34:28.384435 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:34:28.384435 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:28.384435 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:28.384435 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:34:28.384435 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:28.384862 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:34:28.384862 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:34:28.384862 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:34:28.384862 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:34:28.384862 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:34:28.384862 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:28.384862 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:28.384862 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:28.384862 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:34:28.384862 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:28.384862 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:28.384862 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:28.384862 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:34:28.384862 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:28.384862 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:34:28.384862 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:34:28.384862 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:28.384862 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:34:28.385334 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:28.385334 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:34:28.385334 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:34:28.385334 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:28.385334 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:34:28.385334 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:28.385334 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:34:28.389277 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:34:28.390493 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:34:28.390493 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:34:28.390493 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:34:28.390947 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:34:28.391283 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:28.391980 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:34:28.391980 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:28.391980 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:34:28.391980 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:34:28.391980 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:28.391980 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:34:28.391980 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:28.391980 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:34:28.395178 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:28.395178 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:34:28.395178 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:34:28.395178 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:34:28.395178 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:28.395178 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:34:28.395178 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:28.395178 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:34:28.395178 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:28.395178 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:34:28.395178 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:28.395178 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:34:28.395178 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:28.395178 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:34:28.395178 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:34:28.395664 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:28.395664 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:34:28.395664 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:28.395664 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:34:28.395664 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:34:28.395664 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:28.395664 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:34:28.395664 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:28.395664 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:34:28.395664 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:28.395664 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:34:28.395664 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:28.395664 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:34:28.395664 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:34:28.396119 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:34:28.396119 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:34:28.396119 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:34:28.396119 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:34:28.396119 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:28.396119 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:28.396119 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:28.396119 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:34:28.396119 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:28.396119 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:28.396119 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:28.396119 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:34:28.396119 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:28.396586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:34:28.396586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:34:28.396586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:28.396586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:34:28.396586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:28.396586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:34:28.396586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:34:28.396586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:28.396586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:34:28.396586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:28.396586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:34:28.397270 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:28.397270 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:34:28.397270 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:34:28.397270 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:34:28.397270 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:34:28.397270 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:34:28.397270 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:28.397270 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:34:28.397270 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:34:28.397270 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:28.397270 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:34:28.397270 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:34:28.397270 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:28.397270 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:28.397270 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:34:28.397270 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:28.397270 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:34:28.397641 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:34:28.397641 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:34:28.397641 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:34:28.397641 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:28.397641 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:28.397641 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:28.397641 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:34:28.397641 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:28.397641 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:28.397641 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:28.397641 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:34:28.397641 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:28.397641 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:34:28.397641 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:34:28.397641 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:28.397641 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:34:28.397641 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:28.397641 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:34:28.397641 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:34:28.398287 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:28.398287 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:34:28.398287 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:28.398287 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:34:28.398985 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:34:28.399791 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:34:28.400004 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:34:28.400289 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:34:28.402851 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:34:28.403189 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:34:28.403313 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:28.403921 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:34:28.403921 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:28.403921 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:34:28.403921 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:34:28.403921 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:28.403921 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:34:28.403921 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:28.403921 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:34:28.405033 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:28.405033 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:34:28.405033 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:28.405033 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:34:28.405033 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:34:28.405033 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:34:28.405033 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:34:28.405033 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:28.405033 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:28.405033 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:28.405033 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:34:28.405033 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:28.405264 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:28.405264 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:28.405264 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:34:28.405264 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:28.405264 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:34:28.405264 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:34:28.405264 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:28.405264 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:34:28.405264 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:28.405264 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:34:28.405264 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:34:28.405264 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:28.405264 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:34:28.405717 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:28.405717 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:34:28.406164 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:28.406164 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:34:28.406164 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:34:28.406164 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:34:28.406164 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:34:28.406164 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:34:28.406164 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:28.406164 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:34:28.406164 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:34:28.406164 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:28.406164 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:34:28.406164 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:34:28.406164 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:28.406164 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:28.406164 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:34:28.406164 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:28.406729 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:34:28.406729 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:34:28.406729 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:28.406729 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:28.406729 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:28.406729 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:34:28.406729 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:28.406729 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:28.406729 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:28.406729 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:34:28.406729 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:28.406729 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:34:28.406729 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:34:28.406729 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:28.406729 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:34:28.406729 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:28.406729 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:34:28.406729 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:34:28.406729 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:28.407316 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:34:28.407316 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:28.407316 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:34:32.892495 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:34:32.893257 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:34:32.893257 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:34:32.894630 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:34:32.928804 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:34:32.929111 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:32.930015 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:34:32.930015 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:32.930015 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:34:32.930015 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:34:32.930015 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:32.930015 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:34:32.930015 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:32.930015 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:34:32.930934 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.037 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:32.930934 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:34:32.930934 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:32.930934 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:34:32.930934 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:34:32.930934 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:34:32.930934 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:34:32.930934 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:32.930934 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:32.930934 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:32.930934 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:34:32.930934 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:32.931489 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:32.931489 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:32.931489 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:34:32.931489 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:32.931489 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:34:32.931489 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:34:32.931489 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:32.931489 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:34:32.931489 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:32.931489 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:34:32.931489 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:34:32.931489 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:32.931489 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:34:32.932058 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:32.932058 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:34:32.932058 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:32.932058 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:34:32.932058 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:34:32.932058 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:34:32.932058 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:34:32.932058 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:34:32.932058 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:32.932058 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:34:32.932058 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:34:32.932058 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:32.932058 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:34:32.932058 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:34:32.932058 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:32.932721 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:32.932721 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:34:32.932721 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:32.932721 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:34:32.932721 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:34:32.932721 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:32.932721 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:32.932721 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:32.932721 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:34:32.932721 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:32.932721 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:32.932721 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:32.932721 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:34:32.932721 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:32.932721 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:34:32.932721 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:34:32.932721 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:32.932721 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:34:32.932721 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:32.933603 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:34:32.933603 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:34:32.933603 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:32.933603 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:34:32.933603 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:32.933603 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:34:33.360786 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:34:33.361971 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:34:33.362110 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:34:33.363144 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:34:33.363526 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:33.365312 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:34:33.365312 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:33.365312 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:34:33.365312 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:34:33.365312 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:33.365312 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:34:33.365312 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:33.365312 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:34:33.366439 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:33.366439 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:34:33.366439 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:33.366439 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:34:33.366439 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:34:33.366439 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:34:33.366439 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:34:33.366439 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:33.366439 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:33.366439 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:33.366439 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:34:33.366439 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:33.367753 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:33.367753 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:33.367753 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:34:33.367753 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:33.367753 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:34:33.367753 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:34:33.367753 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:33.367753 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:34:33.367753 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:33.367753 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:34:33.367753 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:34:33.367753 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:33.367753 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:34:33.368779 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:33.368779 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:34:33.368779 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:33.368779 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:34:33.368779 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:34:33.368779 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:34:33.368779 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:34:33.368779 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:34:33.368779 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:33.368779 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:34:33.368779 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:34:33.368779 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:33.368779 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:34:33.368779 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:34:33.368779 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:33.369576 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:33.369576 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:34:33.369576 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:33.369576 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:34:33.369576 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:34:33.369576 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:33.369576 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:33.369576 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:33.369576 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:34:33.369576 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:33.369576 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:33.369576 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:33.369576 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:34:33.369576 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:33.369576 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:34:33.369576 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:34:33.369576 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:33.369576 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:34:33.369576 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:33.370493 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:34:33.370493 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:34:33.370493 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:33.370493 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:34:33.370493 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:33.370493 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:34:33.378518 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:34:33.378973 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:34:33.379243 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:34:33.379472 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:34:33.380083 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:34:33.380515 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:33.381532 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:34:33.381532 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:33.381532 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:34:33.381532 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:34:33.381532 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:33.381532 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:34:33.381532 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:33.381532 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:34:33.382746 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:33.382746 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:34:33.382746 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:34:33.382746 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:34:33.382746 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:33.382746 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:34:33.382746 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:33.382746 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:34:33.382746 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:33.382746 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:34:33.382746 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:33.382746 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:34:33.382746 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:33.382746 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:34:33.382746 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:34:33.383260 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:33.383260 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:34:33.383260 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:33.383260 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:34:33.383260 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:34:33.383260 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:33.383260 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:34:33.383260 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:33.383260 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:34:33.384187 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:33.384187 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:34:33.384187 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:33.384187 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:34:33.384187 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:34:33.384187 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:34:33.384187 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:34:33.384187 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:34:33.384187 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:34:33.384187 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:34:33.384187 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:33.384187 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:33.384740 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:33.384740 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:34:33.384740 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:33.384740 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:33.384740 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:33.384740 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:34:33.384740 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:33.384740 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:34:33.384740 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:34:33.384740 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:33.384740 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:34:33.384740 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:33.384740 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:34:33.384740 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:34:33.385408 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:33.385408 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:34:33.385408 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:33.385408 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:34:33.385408 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:33.385408 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:34:33.385408 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:34:33.385408 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:34:33.385408 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:34:33.385408 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:34:33.385408 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:33.385408 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:34:33.385408 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:34:33.385408 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:33.386575 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:34:33.386575 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:34:33.386575 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:33.386575 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:33.386575 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:34:33.386575 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:33.386575 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:34:33.386575 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:34:33.386575 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:34:33.386575 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:34:33.386575 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:34:33.386575 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:33.386575 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:33.386575 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:33.386575 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:34:33.386575 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:33.386575 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:33.386575 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:33.386575 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:34:33.387861 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:33.387861 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:34:33.387861 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:34:33.387861 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:33.387861 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:34:33.387861 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:33.387861 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:34:33.387861 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:34:33.387861 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:33.387861 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:34:33.387861 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:33.387861 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:34:33.389934 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:34:33.390271 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:34:33.390401 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:34:33.390543 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:34:33.390965 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:34:33.391241 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:33.391855 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:34:33.391855 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:33.391855 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:34:33.391855 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:34:33.391855 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:33.391855 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:34:33.391855 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:33.391855 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:34:33.392927 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:33.392927 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:34:33.392927 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:34:33.392927 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:34:33.392927 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:33.392927 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:34:33.392927 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:33.392927 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:34:33.392927 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:33.392927 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:34:33.392927 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:33.392927 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:34:33.392927 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:33.392927 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:34:33.392927 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:34:33.393503 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:33.393503 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:34:33.393503 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:33.393503 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:34:33.393503 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:34:33.393503 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:33.393503 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:34:33.393503 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:33.393503 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:34:33.393503 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:33.393503 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:34:33.393503 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:33.393503 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:34:33.393503 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:34:33.393955 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:34:33.393955 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:34:33.393955 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:34:33.393955 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:34:33.393955 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:33.393955 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:33.393955 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:33.393955 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:34:33.393955 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:33.393955 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:33.393955 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:33.393955 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:34:33.393955 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:33.394371 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:34:33.394371 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:34:33.394371 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:33.394371 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:34:33.394371 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:33.394371 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:34:33.394371 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:34:33.394371 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:33.394371 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:34:33.394371 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:33.394371 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:34:33.394770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:33.394770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:34:33.394770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:34:33.394770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:34:33.394770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:34:33.394770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:34:33.394770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:33.394770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:34:33.394770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:34:33.394770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:33.394770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:34:33.394770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:34:33.394770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:33.394770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:33.394770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:34:33.394770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:33.394770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:34:33.395336 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:34:33.395336 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:34:33.395336 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:34:33.395336 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:33.395336 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:33.395336 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:33.395336 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:34:33.395336 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:33.395336 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:33.395336 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:33.395336 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:34:33.395336 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:33.395336 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:34:33.395336 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:34:33.395336 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:33.395336 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:34:33.395336 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:33.395336 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:34:33.395336 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:34:33.395877 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:33.395877 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:34:33.395877 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:33.395877 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:34:33.399582 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:34:33.399877 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:34:33.399993 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:34:33.400151 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:34:33.400531 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:34:33.400774 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:34:33.400901 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:33.401482 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:34:33.401482 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:33.401482 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:34:33.401482 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:34:33.401482 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:33.401482 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:34:33.401482 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:33.401482 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:34:33.402338 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:33.402338 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:34:33.402338 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:33.402338 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:34:33.402338 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:34:33.402338 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:34:33.402338 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:34:33.402338 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:33.402338 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:33.402338 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:33.402338 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:34:33.402338 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:33.402724 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:33.402724 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:33.402724 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:34:33.402724 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:33.402724 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:34:33.402724 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:34:33.402724 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:33.402724 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:34:33.402724 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:33.402724 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:34:33.402724 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:34:33.402724 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:33.402724 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:34:33.403153 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:33.403153 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:34:33.403404 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:33.403404 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:34:33.403404 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:34:33.403404 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:34:33.403404 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:34:33.403404 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:34:33.403404 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:33.403404 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:34:33.403404 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:34:33.403404 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:33.403404 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:34:33.403404 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:34:33.403404 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:33.403404 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:33.403404 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:34:33.403404 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:33.403898 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:34:33.403898 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:34:33.403898 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:33.403898 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:33.403898 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:33.403898 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:34:33.403898 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:33.403898 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:33.403898 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:33.403898 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:34:33.403898 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:33.403898 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:34:33.403898 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:34:33.403898 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:33.403898 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:34:33.403898 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:33.403898 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:34:33.403898 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:34:33.403898 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:33.404451 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:34:33.404451 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:33.404451 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:34:37.893288 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:34:37.894038 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:34:37.894268 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:34:37.895752 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:34:37.932042 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:34:37.932348 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:37.933135 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:34:37.933135 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:37.933135 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:34:37.933135 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:34:37.933135 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:37.933135 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:34:37.933135 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:37.933135 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:34:37.934042 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.039 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:37.934042 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:34:37.934042 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:37.934042 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:34:37.934042 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:34:37.934042 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:34:37.934042 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:34:37.934042 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:37.934042 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:37.934042 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:37.934042 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:34:37.934042 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:37.934905 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:37.934905 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:37.934905 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:34:37.934905 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:37.934905 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:34:37.934905 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:34:37.934905 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:37.934905 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:34:37.934905 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:37.934905 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:34:37.934905 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:34:37.934905 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:37.934905 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:34:37.935745 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:37.935745 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:34:37.935745 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:37.935745 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:34:37.935745 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:34:37.935745 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:34:37.935745 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:34:37.935745 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:34:37.935745 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:37.935745 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:34:37.935745 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:34:37.935745 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:37.935745 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:34:37.935745 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:34:37.935745 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:37.936586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:37.936586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:34:37.936586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:37.936586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:34:37.936586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:34:37.936586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:37.936586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:37.936586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:37.936586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:34:37.936586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:37.936586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:37.936586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:37.936586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:34:37.936586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:37.936586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:34:37.936586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:34:37.936586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:37.936586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:34:37.936586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:37.937684 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:34:37.937684 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:34:37.937684 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:37.937684 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:34:37.937684 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:37.937684 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:34:38.363080 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:34:38.363080 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:34:38.363398 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:34:38.364418 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:34:38.364810 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:38.365749 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:34:38.365749 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:38.365749 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:34:38.365749 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:34:38.365749 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:38.365749 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:34:38.365749 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:38.365749 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:34:38.367007 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:38.367007 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:34:38.367007 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:38.367007 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:34:38.367007 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:34:38.367007 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:34:38.367007 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:34:38.367007 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:38.367007 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:38.367007 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:38.367007 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:34:38.367007 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:38.367555 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:38.367555 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:38.367555 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:34:38.367555 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:38.367555 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:34:38.367555 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:34:38.367555 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:38.367555 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:34:38.367555 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:38.367555 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:34:38.367555 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:34:38.367555 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:38.367555 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:34:38.368099 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:38.368099 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:34:38.368631 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:38.368631 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:34:38.368631 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:34:38.368631 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:34:38.368631 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:34:38.368631 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:34:38.368631 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:38.368631 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:34:38.368631 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:34:38.368631 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:38.368631 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:34:38.368631 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:34:38.368631 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:38.368631 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:38.368631 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:34:38.368631 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:38.369412 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:34:38.369412 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:34:38.369412 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:38.369412 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:38.369412 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:38.369412 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:34:38.369412 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:38.369412 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:38.369412 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:38.369412 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:34:38.369412 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:38.369412 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:34:38.369412 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:34:38.369412 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:38.369412 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:34:38.369412 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:38.369412 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:34:38.369412 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:34:38.369412 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:38.370214 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:34:38.370214 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:38.370214 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:34:38.378968 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:34:38.379294 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:34:38.379485 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:34:38.379823 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:34:38.380630 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:34:38.381096 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:38.382254 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:34:38.382254 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:38.382254 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:34:38.382254 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:34:38.382254 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:38.382254 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:34:38.382254 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:38.382254 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:34:38.383569 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:38.383569 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:34:38.383569 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:34:38.383569 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:34:38.383569 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:38.383569 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:34:38.383569 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:38.383569 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:34:38.383569 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:38.383569 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:34:38.383569 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:38.383569 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:34:38.383569 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:38.383569 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:34:38.383569 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:34:38.384272 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:38.384272 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:34:38.384272 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:38.384272 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:34:38.384272 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:34:38.384272 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:38.384272 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:34:38.384272 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:38.384272 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:34:38.384999 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:38.384999 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:34:38.384999 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:38.384999 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:34:38.384999 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:34:38.384999 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:34:38.384999 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:34:38.384999 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:34:38.384999 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:34:38.384999 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:34:38.384999 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:38.384999 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:38.385635 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:38.385635 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:34:38.385635 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:38.385635 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:38.385635 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:38.385635 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:34:38.385635 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:38.385635 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:34:38.385635 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:34:38.385635 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:38.385635 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:34:38.385635 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:38.385635 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:34:38.385635 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:34:38.386466 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:38.386466 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:34:38.386466 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:38.386466 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:34:38.387022 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:38.387022 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:34:38.387022 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:34:38.387022 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:34:38.387022 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:34:38.387022 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:34:38.387022 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:38.387022 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:34:38.387022 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:34:38.387022 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:38.387022 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:34:38.387022 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:34:38.387022 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:38.387022 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:38.387022 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:34:38.387022 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:38.387783 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:34:38.387783 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:34:38.387783 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:34:38.387783 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:34:38.387783 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:34:38.387783 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:38.387783 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:38.387783 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:38.387783 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:34:38.387783 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:38.387783 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:38.387783 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:38.387783 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:34:38.387783 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:38.387783 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:34:38.387783 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:34:38.387783 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:38.387783 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:34:38.388678 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:38.388678 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:34:38.388678 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:34:38.388678 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:38.388678 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:34:38.388678 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:38.388678 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:34:38.390402 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:34:38.390805 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:34:38.390930 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:34:38.391257 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:34:38.391672 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:34:38.391923 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:38.392600 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:34:38.392600 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:38.392600 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:34:38.392600 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:34:38.392600 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:38.392600 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:34:38.392600 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:38.392600 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:34:38.393510 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:38.393510 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:34:38.393510 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:34:38.393510 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:34:38.393510 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:38.393510 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:34:38.393510 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:38.393510 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:34:38.393510 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:38.393510 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:34:38.393510 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:38.393510 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:34:38.393510 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:38.393510 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:34:38.393510 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:34:38.393962 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:38.393962 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:34:38.393962 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:38.393962 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:34:38.393962 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:34:38.393962 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:38.393962 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:34:38.393962 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:38.393962 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:34:38.394685 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:38.394685 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:34:38.394685 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:38.394685 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:34:38.394685 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:34:38.394685 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:34:38.394685 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:34:38.394685 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:34:38.394685 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:34:38.394685 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:38.394685 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:38.395011 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:38.395011 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:34:38.395011 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:38.395011 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:38.395011 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:38.395011 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:34:38.395011 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:38.395011 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:34:38.395011 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:34:38.395011 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:38.395011 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:34:38.395011 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:38.395011 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:34:38.395011 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:34:38.395461 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:38.395461 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:34:38.395461 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:38.395461 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:34:38.396087 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:38.396087 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:34:38.396087 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:34:38.396087 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:34:38.396087 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:34:38.396087 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:34:38.396087 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:38.396087 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:34:38.396087 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:34:38.396087 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:38.396087 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:34:38.396087 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:34:38.396087 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:38.396087 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:38.396087 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:34:38.396087 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:38.396087 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:34:38.396645 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:34:38.396645 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:34:38.396645 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:34:38.396645 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:38.396645 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:38.396645 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:38.396645 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:34:38.396645 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:38.396645 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:38.396645 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:38.396645 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:34:38.396645 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:38.396645 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:34:38.396645 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:34:38.396645 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:38.396645 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:34:38.396645 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:38.396645 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:34:38.396645 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:34:38.397157 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:38.397157 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:34:38.397157 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:38.397157 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:34:38.400026 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:34:38.400462 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:34:38.400627 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:34:38.400825 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:34:38.401298 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:34:38.401716 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:34:38.401909 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:38.402642 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:34:38.402642 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:38.402642 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:34:38.402642 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:34:38.402642 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:38.402642 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:34:38.402642 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:38.402642 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:34:38.403629 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:38.403629 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:34:38.403629 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:38.403629 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:34:38.403629 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:34:38.403629 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:34:38.403629 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:34:38.403629 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:38.403629 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:38.403629 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:38.403629 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:34:38.403629 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:38.403999 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:38.403999 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:38.403999 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:34:38.403999 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:38.403999 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:34:38.403999 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:34:38.403999 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:38.403999 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:34:38.403999 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:38.403999 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:34:38.403999 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:34:38.403999 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:38.403999 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:34:38.404433 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:38.404433 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:34:38.404923 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:38.404923 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:34:38.404923 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:34:38.404923 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:34:38.404923 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:34:38.404923 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:34:38.404923 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:38.404923 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:34:38.404923 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:34:38.404923 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:38.404923 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:34:38.404923 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:34:38.404923 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:38.404923 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:38.404923 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:34:38.404923 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:38.405544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:34:38.405544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:34:38.405544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:38.405544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:38.405544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:38.405544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:34:38.405544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:38.405544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:38.405544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:38.405544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:34:38.405544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:38.405544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:34:38.405544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:34:38.405544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:38.405544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:34:38.405544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:38.405544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:34:38.405544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:34:38.405544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:38.406083 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:34:38.406083 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:38.406083 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:34:42.894147 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:34:42.895128 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:34:42.895268 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:34:42.896356 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:34:42.898743 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG dbcounter [-] [106123] Writing DB stats neutron:SELECT=520,neutron:INSERT=1,neutron:UPDATE=2,neutron:DELETE=1 {{(pid=106123) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} May 18 18:34:42.929315 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:34:42.929766 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:42.930618 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:34:42.930618 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:42.930618 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:34:42.930618 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:34:42.930618 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:42.930618 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:34:42.930618 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:42.930618 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:34:42.931314 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.035 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:42.931314 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:34:42.931314 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:42.931314 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:34:42.931314 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:34:42.931314 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:34:42.931314 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:34:42.931314 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:42.931314 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:42.931314 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:42.931314 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:34:42.931314 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:42.932133 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:42.932133 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:42.932133 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:34:42.932133 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:42.932133 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:34:42.932133 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:34:42.932133 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:42.932133 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:34:42.932133 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:42.932133 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:34:42.932133 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:34:42.932133 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:42.932133 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:34:42.932929 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:42.932929 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:34:42.932929 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:42.932929 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:34:42.932929 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:34:42.932929 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:34:42.932929 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:34:42.932929 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:34:42.932929 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:42.932929 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:34:42.932929 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:34:42.932929 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:42.932929 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:34:42.932929 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:34:42.932929 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:42.933671 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:42.933671 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:34:42.933671 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:42.933671 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:34:42.933671 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:34:42.933671 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:42.933671 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:42.933671 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:42.933671 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:34:42.933671 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:42.933671 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:42.933671 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:42.933671 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:34:42.933671 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:42.933671 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:34:42.933671 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:34:42.933671 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:42.933671 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:34:42.933671 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:42.934876 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:34:42.934876 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:34:42.934876 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:42.934876 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:34:42.934876 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:42.934876 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:34:43.363085 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:34:43.363874 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:34:43.363988 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:34:43.365096 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:34:43.365588 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:43.366610 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:34:43.366610 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:43.366610 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:34:43.366610 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:34:43.366610 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:43.366610 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:34:43.366610 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:43.366610 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:34:43.367992 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:43.367992 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:34:43.367992 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:43.367992 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:34:43.367992 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:34:43.367992 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:34:43.367992 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:34:43.367992 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:43.367992 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:43.367992 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:43.367992 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:34:43.367992 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:43.368668 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:43.368668 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:43.368668 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:34:43.368668 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:43.368668 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:34:43.368668 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:34:43.368668 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:43.368668 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:34:43.368668 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:43.368668 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:34:43.368668 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:34:43.368668 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:43.368668 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:34:43.369443 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:43.369443 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:34:43.369443 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:43.369443 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:34:43.369443 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:34:43.369443 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:34:43.369443 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:34:43.369443 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:34:43.369443 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:43.369443 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:34:43.369443 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:34:43.369443 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:43.369443 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:34:43.369443 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:34:43.369443 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:43.370777 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:43.370777 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:34:43.370777 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:43.370777 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:34:43.370777 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:34:43.370777 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:43.370777 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:43.370777 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:43.370777 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:34:43.370777 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:43.370777 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:43.370777 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:43.370777 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:34:43.370777 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:43.370777 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:34:43.370777 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:34:43.370777 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:43.370777 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:34:43.370777 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:43.371895 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:34:43.371895 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:34:43.371895 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:43.371895 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:34:43.371895 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:43.371895 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:34:43.379580 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:34:43.379947 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:34:43.380069 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:34:43.380201 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:34:43.381153 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:34:43.381621 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:43.382683 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:34:43.382683 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:43.382683 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:34:43.382683 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:34:43.382683 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:43.382683 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:34:43.382683 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:43.382683 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:34:43.384105 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:43.384105 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:34:43.384105 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:34:43.384105 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:34:43.384105 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:43.384105 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:34:43.384105 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:43.384105 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:34:43.384105 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:43.384105 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:34:43.384105 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:43.384105 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:34:43.384105 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:43.384105 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:34:43.384105 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:34:43.384646 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:43.384646 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:34:43.384646 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:43.384646 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:34:43.384646 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:34:43.384646 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:43.384646 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:34:43.384646 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:43.384646 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:34:43.385527 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:43.385527 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:34:43.385527 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:43.385527 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:34:43.385527 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:34:43.385527 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:34:43.385527 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:34:43.385527 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:34:43.385527 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:34:43.385527 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:34:43.385527 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:43.385527 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:43.386120 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:43.386120 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:34:43.386120 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:43.386120 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:43.386120 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:43.386120 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:34:43.386120 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:43.386120 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:34:43.386120 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:34:43.386120 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:43.386120 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:34:43.386120 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:43.386120 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:34:43.386120 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:34:43.386685 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:43.386685 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:34:43.386685 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:43.386685 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:34:43.387598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:43.387598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:34:43.387598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:34:43.387598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:34:43.387598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:34:43.387598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:34:43.387598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:43.387598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:34:43.387598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:34:43.387598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:43.387598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:34:43.387598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:34:43.387598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:43.387598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:43.387598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:34:43.387598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:43.388212 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:34:43.388212 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:34:43.388212 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:34:43.388212 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:34:43.388212 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:34:43.388212 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:43.388212 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:43.388212 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:43.388212 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:34:43.388212 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:43.388212 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:43.388212 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:43.388212 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:34:43.388212 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:43.388212 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:34:43.388212 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:34:43.388212 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:43.388212 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:34:43.388934 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:43.388934 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:34:43.388934 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:34:43.388934 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:43.388934 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:34:43.388934 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:43.388934 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:34:43.390913 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:34:43.391348 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:34:43.391549 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:34:43.391765 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:34:43.392178 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:34:43.392434 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:43.392993 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:34:43.392993 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:43.392993 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:34:43.392993 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:34:43.392993 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:43.392993 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:34:43.392993 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:43.392993 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:34:43.393853 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:43.393853 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:34:43.393853 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:34:43.393853 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:34:43.393853 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:43.393853 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:34:43.393853 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:43.393853 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:34:43.393853 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:43.393853 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:34:43.393853 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:43.393853 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:34:43.393853 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:43.393853 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:34:43.393853 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:34:43.394530 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:43.394530 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:34:43.394530 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:43.394530 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:34:43.394530 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:34:43.394530 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:43.394530 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:34:43.394530 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:43.394530 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:34:43.394909 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:43.394909 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:34:43.394909 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:43.394909 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:34:43.394909 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:34:43.394909 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:34:43.394909 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:34:43.394909 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:34:43.394909 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:34:43.394909 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:43.394909 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:43.395401 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:43.395401 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:34:43.395401 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:43.395401 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:43.395401 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:43.395401 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:34:43.395401 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:43.395401 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:34:43.395401 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:34:43.395401 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:43.395401 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:34:43.395401 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:43.395401 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:34:43.395401 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:34:43.395950 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:43.395950 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:34:43.395950 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:43.395950 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:34:43.396148 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:43.396148 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:34:43.396148 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:34:43.396148 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:34:43.396148 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:34:43.396148 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:34:43.396148 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:43.396148 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:34:43.396148 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:34:43.396148 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:43.396148 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:34:43.396148 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:34:43.396148 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:43.396148 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:43.396148 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:34:43.396148 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:43.396148 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:34:43.396860 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:34:43.396860 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:34:43.396860 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:34:43.396860 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:43.396860 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:43.396860 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:43.396860 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:34:43.396860 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:43.396860 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:43.396860 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:43.396860 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:34:43.396860 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:43.396860 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:34:43.396860 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:34:43.396860 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:43.396860 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:34:43.396860 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:43.396860 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:34:43.396860 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:34:43.398109 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:43.398109 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:34:43.398109 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:43.398109 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:34:43.400633 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:34:43.401021 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:34:43.401162 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:34:43.401385 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:34:43.401888 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:34:43.402227 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:34:43.402445 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:43.403218 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:34:43.403218 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:43.403218 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:34:43.403218 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:34:43.403218 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:43.403218 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:34:43.403218 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:43.403218 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:34:43.404194 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:43.404194 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:34:43.404194 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:43.404194 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:34:43.404194 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:34:43.404194 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:34:43.404194 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:34:43.404194 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:43.404194 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:43.404194 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:43.404194 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:34:43.404194 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:43.404819 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:43.404819 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:43.404819 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:34:43.404819 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:43.404819 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:34:43.404819 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:34:43.404819 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:43.404819 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:34:43.404819 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:43.404819 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:34:43.404819 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:34:43.404819 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:43.404819 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:34:43.405459 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:43.405459 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:34:43.405459 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:43.405459 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:34:43.405459 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:34:43.405459 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:34:43.405459 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:34:43.405459 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:34:43.405459 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:43.405459 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:34:43.405459 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:34:43.405459 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:43.405459 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:34:43.405459 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:34:43.405459 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:43.406232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:43.406232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:34:43.406232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:43.406232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:34:43.406232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:34:43.406232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:43.406232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:43.406232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:43.406232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:34:43.406232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:43.406232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:43.406232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:43.406232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:34:43.406232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:43.406232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:34:43.406232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:34:43.406232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:43.406232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:34:43.406232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:43.407039 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:34:43.407039 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:34:43.407039 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:43.407039 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:34:43.407039 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:43.407039 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:34:47.895251 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:34:47.896454 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:34:47.896454 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:34:47.897385 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:34:47.938676 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:34:47.939193 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:47.940427 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:34:47.940427 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:47.940427 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:34:47.940427 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:34:47.940427 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:47.940427 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:34:47.940427 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:47.940427 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:34:47.941294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.044 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:47.941294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:34:47.941294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:47.941294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:34:47.941294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:34:47.941294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:34:47.941294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:34:47.941294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:47.941294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:47.941294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:47.941294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:34:47.941294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:47.941912 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:47.941912 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:47.941912 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:34:47.941912 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:47.941912 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:34:47.941912 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:34:47.941912 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:47.941912 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:34:47.941912 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:47.941912 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:34:47.941912 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:34:47.941912 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:47.941912 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:34:47.942253 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:47.942253 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:34:47.942253 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:47.942253 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:34:47.942253 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:34:47.942253 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:34:47.942253 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:34:47.942253 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:34:47.942253 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:47.942253 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:34:47.942253 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:34:47.942253 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:47.942253 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:34:47.942253 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:34:47.942253 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:47.942694 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:47.942694 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:34:47.942694 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:47.942694 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:34:47.942694 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:34:47.942694 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:47.942694 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:47.942694 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:47.942694 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:34:47.942694 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:47.942694 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:47.942694 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:47.942694 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:34:47.942694 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:47.942694 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:34:47.942694 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:34:47.942694 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:47.942694 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:34:47.942694 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:47.943212 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:34:47.943212 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:34:47.943212 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:47.943212 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:34:47.943212 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:47.943212 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:34:48.364147 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:34:48.364895 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:34:48.365087 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:34:48.366128 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:34:48.366509 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:48.367518 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:34:48.367518 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:48.367518 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:34:48.367518 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:34:48.367518 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:48.367518 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:34:48.367518 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:48.367518 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:34:48.369302 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:48.369302 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:34:48.369302 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:48.369302 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:34:48.369302 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:34:48.369302 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:34:48.369302 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:34:48.369302 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:48.369302 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:48.369302 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:48.369302 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:34:48.369302 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:48.370172 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:48.370172 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:48.370172 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:34:48.370172 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:48.370172 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:34:48.370172 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:34:48.370172 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:48.370172 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:34:48.370172 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:48.370172 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:34:48.370172 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:34:48.370172 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:48.370172 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:34:48.371020 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:48.371020 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:34:48.371899 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:48.371899 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:34:48.371899 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:34:48.371899 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:34:48.371899 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:34:48.371899 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:34:48.371899 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:48.371899 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:34:48.371899 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:34:48.371899 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:48.371899 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:34:48.371899 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:34:48.371899 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:48.371899 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:48.371899 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:34:48.371899 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:48.372415 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:34:48.372415 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:34:48.372415 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:48.372415 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:48.372415 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:48.372415 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:34:48.372415 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:48.372415 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:48.372415 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:48.372415 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:34:48.372415 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:48.372415 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:34:48.372415 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:34:48.372415 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:48.372415 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:34:48.372415 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:48.372415 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:34:48.372415 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:34:48.372415 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:48.372916 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:34:48.372916 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:48.372916 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:34:48.380100 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:34:48.380462 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:34:48.380615 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:34:48.380786 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:34:48.381283 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:34:48.381641 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:48.382314 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:34:48.382314 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:48.382314 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:34:48.382314 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:34:48.382314 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:48.382314 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:34:48.382314 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:48.382314 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:34:48.383176 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:48.383176 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:34:48.383176 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:34:48.383176 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:34:48.383176 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:48.383176 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:34:48.383176 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:48.383176 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:34:48.383176 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:48.383176 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:34:48.383176 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:48.383176 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:34:48.383176 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:48.383176 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:34:48.383176 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:34:48.383853 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:48.383853 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:34:48.383853 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:48.383853 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:34:48.383853 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:34:48.383853 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:48.383853 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:34:48.383853 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:48.383853 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:34:48.384406 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:48.384406 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:34:48.384406 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:48.384406 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:34:48.384406 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:34:48.384406 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:34:48.384406 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:34:48.384406 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:34:48.384406 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:34:48.384406 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:34:48.384406 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:48.384406 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:48.384984 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:48.384984 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:34:48.384984 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:48.384984 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:48.384984 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:48.384984 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:34:48.384984 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:48.384984 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:34:48.384984 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:34:48.384984 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:48.384984 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:34:48.384984 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:48.384984 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:34:48.384984 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:34:48.385988 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:48.385988 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:34:48.385988 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:48.385988 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:34:48.386600 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:48.386600 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:34:48.386600 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:34:48.386600 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:34:48.386600 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:34:48.386600 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:34:48.386600 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:48.386600 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:34:48.386600 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:34:48.386600 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:48.386600 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:34:48.386600 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:34:48.386600 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:48.386600 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:48.386600 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:34:48.386600 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:48.387396 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:34:48.387396 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:34:48.387396 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:34:48.387396 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:34:48.387396 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:34:48.387396 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:48.387396 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:48.387396 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:48.387396 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:34:48.387396 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:48.387396 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:48.387396 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:48.387396 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:34:48.387396 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:48.387396 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:34:48.387396 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:34:48.387396 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:48.387396 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:34:48.388248 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:48.388248 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:34:48.388248 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:34:48.388248 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:48.388248 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:34:48.388248 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:48.388248 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:34:48.391402 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:34:48.391878 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:34:48.392030 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:34:48.392211 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:34:48.392679 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:34:48.393046 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:48.393743 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:34:48.393743 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:48.393743 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:34:48.393743 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:34:48.393743 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:48.393743 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:34:48.393743 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:48.393743 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:34:48.394577 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:48.394577 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:34:48.394577 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:34:48.394577 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:34:48.394577 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:48.394577 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:34:48.394577 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:48.394577 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:34:48.394577 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:48.394577 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:34:48.394577 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:48.394577 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:34:48.394577 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:48.394577 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:34:48.394577 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:34:48.395193 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:48.395193 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:34:48.395193 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:48.395193 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:34:48.395193 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:34:48.395193 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:48.395193 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:34:48.395193 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:48.395193 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:34:48.395822 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:48.395822 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:34:48.395822 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:48.395822 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:34:48.395822 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:34:48.395822 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:34:48.395822 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:34:48.395822 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:34:48.395822 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:34:48.395822 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:48.395822 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:48.396292 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:48.396292 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:34:48.396292 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:48.396292 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:48.396292 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:48.396292 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:34:48.396292 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:48.396292 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:34:48.396292 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:34:48.396292 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:48.396292 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:34:48.396292 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:48.396292 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:34:48.396292 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:34:48.396909 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:48.396909 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:34:48.396909 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:48.396909 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:34:48.397291 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:48.397291 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:34:48.397291 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:34:48.397291 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:34:48.397291 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:34:48.397291 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:34:48.397291 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:48.397291 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:34:48.397291 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:34:48.397291 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:48.397291 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:34:48.397291 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:34:48.397291 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:48.397291 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:48.397291 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:34:48.397291 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:48.397291 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:34:48.397847 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:34:48.397847 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:34:48.397847 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:34:48.397847 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:48.397847 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:48.397847 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:48.397847 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:34:48.397847 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:48.397847 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:48.397847 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:48.397847 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:34:48.397847 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:48.397847 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:34:48.397847 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:34:48.397847 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:48.397847 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:34:48.397847 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:48.397847 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:34:48.397847 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:34:48.398303 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:48.398303 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:34:48.398303 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:48.398303 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:34:48.401147 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:34:48.401502 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:34:48.401674 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:34:48.401883 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:34:48.402337 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:34:48.402728 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:34:48.402920 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:48.403684 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:34:48.403684 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:48.403684 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:34:48.403684 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:34:48.403684 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:48.403684 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:34:48.403684 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:48.403684 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:34:48.404609 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:48.404609 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:34:48.404609 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:48.404609 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:34:48.404609 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:34:48.404609 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:34:48.404609 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:34:48.404609 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:48.404609 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:48.404609 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:48.404609 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:34:48.404609 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:48.405131 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:48.405131 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:48.405131 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:34:48.405131 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:48.405131 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:34:48.405131 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:34:48.405131 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:48.405131 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:34:48.405131 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:48.405131 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:34:48.405131 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:34:48.405131 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:48.405131 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:34:48.405871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:48.405871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:34:48.406121 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:48.406121 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:34:48.406121 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:34:48.406121 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:34:48.406121 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:34:48.406121 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:34:48.406121 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:48.406121 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:34:48.406121 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:34:48.406121 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:48.406121 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:34:48.406121 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:34:48.406121 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:48.406121 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:48.406121 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:34:48.406121 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:48.406631 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:34:48.406631 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:34:48.406631 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:48.406631 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:48.406631 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:48.406631 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:34:48.406631 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:48.406631 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:48.406631 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:48.406631 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:34:48.406631 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:48.406631 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:34:48.406631 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:34:48.406631 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:48.406631 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:34:48.406631 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:48.406631 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:34:48.406631 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:34:48.406631 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:48.407144 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:34:48.407144 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:48.407144 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:34:52.896036 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:34:52.897044 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:34:52.897044 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:34:52.898247 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:34:52.926952 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:34:52.927554 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:52.928786 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:34:52.928786 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:52.928786 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:34:52.928786 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:34:52.928786 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:52.928786 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:34:52.928786 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:52.928786 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:34:52.930432 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.032 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:52.930432 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:34:52.930432 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:52.930432 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:34:52.930432 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:34:52.930432 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:34:52.930432 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:34:52.930432 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:52.930432 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:52.930432 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:52.930432 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:34:52.930432 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:52.931089 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:52.931089 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:52.931089 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:34:52.931089 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:52.931089 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:34:52.931089 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:34:52.931089 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:52.931089 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:34:52.931089 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:52.931089 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:34:52.931089 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:34:52.931089 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:52.931089 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:34:52.931958 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:52.931958 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:34:52.932281 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:52.932281 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:34:52.932281 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:34:52.932281 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:34:52.932281 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:34:52.932281 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:34:52.932281 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:52.932281 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:34:52.932281 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:34:52.932281 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:52.932281 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:34:52.932281 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:34:52.932281 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:52.932281 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:52.932281 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:34:52.932281 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:52.933903 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:34:52.933903 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:34:52.933903 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:52.933903 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:52.933903 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:52.933903 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:34:52.933903 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:52.933903 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:52.933903 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:52.933903 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:34:52.933903 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:52.933903 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:34:52.933903 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:34:52.933903 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:52.933903 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:34:52.933903 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:52.933903 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:34:52.933903 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:34:52.933903 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:52.934734 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:34:52.934734 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:52.934734 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:34:53.364915 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:34:53.365776 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:34:53.366032 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:34:53.367171 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:34:53.367588 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:53.368614 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:34:53.368614 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:53.368614 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:34:53.368614 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:34:53.368614 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:53.368614 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:34:53.368614 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:53.368614 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:34:53.370350 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:53.370350 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:34:53.370350 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:53.370350 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:34:53.370350 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:34:53.370350 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:34:53.370350 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:34:53.370350 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:53.370350 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:53.370350 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:53.370350 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:34:53.370350 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:53.370693 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:53.370693 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:53.370693 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:34:53.370693 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:53.370693 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:34:53.370693 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:34:53.370693 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:53.370693 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:34:53.370693 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:53.370693 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:34:53.370693 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:34:53.370693 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:53.370693 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:34:53.371010 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:53.371010 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:34:53.372305 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:53.372305 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:34:53.372305 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:34:53.372305 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:34:53.372305 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:34:53.372305 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:34:53.372305 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:53.372305 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:34:53.372305 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:34:53.372305 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:53.372305 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:34:53.372305 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:34:53.372305 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:53.372305 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:53.372305 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:34:53.372305 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:53.372728 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:34:53.372728 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:34:53.372728 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:53.372728 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:53.372728 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:53.372728 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:34:53.372728 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:53.372728 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:53.372728 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:53.372728 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:34:53.372728 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:53.372728 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:34:53.372728 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:34:53.372728 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:53.372728 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:34:53.372728 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:53.372728 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:34:53.372728 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:34:53.372728 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:53.373182 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:34:53.373182 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:53.373182 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:34:53.380634 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:34:53.381115 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:34:53.381368 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:34:53.381618 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:34:53.382188 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:34:53.382494 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:53.383294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:34:53.383294 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:53.383294 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:34:53.383294 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:34:53.383294 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:53.383294 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:34:53.383294 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:53.383294 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:34:53.384678 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:53.384678 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:34:53.384678 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:34:53.384678 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:34:53.384678 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:53.384678 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:34:53.384678 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:53.384678 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:34:53.384678 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:53.384678 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:34:53.384678 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:53.384678 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:34:53.384678 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:53.384678 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:34:53.384678 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:34:53.385178 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:53.385178 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:34:53.385178 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:53.385178 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:34:53.385178 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:34:53.385178 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:53.385178 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:34:53.385178 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:53.385178 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:34:53.386186 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:53.386186 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:34:53.386186 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:53.386186 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:34:53.386186 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:34:53.386186 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:34:53.386186 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:34:53.386186 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:34:53.386186 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:34:53.386186 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:34:53.386186 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:53.386186 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:53.386518 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:53.386518 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:34:53.386518 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:53.386518 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:53.386518 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:53.386518 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:34:53.386518 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:53.386518 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:34:53.386518 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:34:53.386518 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:53.386518 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:34:53.386518 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:53.386518 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:34:53.386518 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:34:53.386878 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:53.386878 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:34:53.386878 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:53.386878 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:34:53.387241 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:53.387241 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:34:53.387241 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:34:53.387241 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:34:53.387241 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:34:53.387241 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:34:53.387241 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:53.387241 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:34:53.387241 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:34:53.387241 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:53.387241 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:34:53.387241 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:34:53.387241 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:53.387241 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:53.387241 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:34:53.387241 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:53.387810 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:34:53.387810 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:34:53.387810 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:34:53.387810 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:34:53.387810 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:34:53.387810 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:53.387810 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:53.387810 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:53.387810 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:34:53.387810 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:53.387810 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:53.387810 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:53.387810 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:34:53.387810 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:53.387810 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:34:53.387810 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:34:53.387810 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:53.387810 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:34:53.388306 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:53.388306 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:34:53.388306 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:34:53.388306 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:53.388306 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:34:53.388306 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:53.388306 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:34:53.391924 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:34:53.392299 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:34:53.392539 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:34:53.392767 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:34:53.393765 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:34:53.394259 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:53.395460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:34:53.395460 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:53.395460 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:34:53.395460 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:34:53.395460 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:53.395460 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:34:53.395460 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:53.395460 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:34:53.396706 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:53.396706 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:34:53.396706 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:34:53.396706 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:34:53.396706 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:53.396706 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:34:53.396706 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:53.396706 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:34:53.396706 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:53.396706 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:34:53.396706 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:53.396706 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:34:53.396706 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:53.396706 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:34:53.396706 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:34:53.397096 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:53.397096 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:34:53.397096 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:53.397096 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:34:53.397096 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:34:53.397096 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:53.397096 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:34:53.397096 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:53.397096 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:34:53.398160 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:53.398160 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:34:53.398160 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:53.398160 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:34:53.398160 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:34:53.398160 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:34:53.398160 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:34:53.398160 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:34:53.398160 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:34:53.398160 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:53.398160 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:53.398499 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:53.398499 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:34:53.398499 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:53.398499 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:53.398499 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:53.398499 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:34:53.398499 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:53.398499 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:34:53.398499 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:34:53.398499 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:53.398499 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:34:53.398499 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:53.398499 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:34:53.398499 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:34:53.398824 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:53.398824 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:34:53.398824 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:53.398824 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:34:53.399889 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:53.399889 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:34:53.399889 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:34:53.399889 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:34:53.399889 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:34:53.399889 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:34:53.399889 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:53.399889 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:34:53.399889 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:34:53.399889 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:53.399889 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:34:53.399889 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:34:53.399889 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:53.399889 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:53.399889 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:34:53.399889 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:53.399889 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:34:53.400350 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:34:53.400350 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:34:53.400350 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:34:53.400350 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:53.400350 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:53.400350 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:53.400350 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:34:53.400350 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:53.400350 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:53.400350 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:53.400350 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:34:53.400350 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:53.400350 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:34:53.400350 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:34:53.400350 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:53.400350 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:34:53.400350 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:53.400350 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:34:53.400350 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:34:53.400812 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:53.400812 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:34:53.400812 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:53.400812 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:34:53.401699 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:34:53.402031 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:34:53.402182 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:34:53.402432 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:34:53.403441 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:34:53.403670 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:34:53.403804 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:53.404406 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:34:53.404406 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:53.404406 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:34:53.404406 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:34:53.404406 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:53.404406 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:34:53.404406 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:53.404406 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:34:53.405339 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:53.405339 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:34:53.405339 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:53.405339 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:34:53.405339 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:34:53.405339 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:34:53.405339 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:34:53.405339 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:53.405339 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:53.405339 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:53.405339 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:34:53.405339 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:53.405860 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:53.405860 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:53.405860 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:34:53.405860 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:53.405860 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:34:53.405860 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:34:53.405860 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:53.405860 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:34:53.405860 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:53.405860 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:34:53.405860 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:34:53.405860 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:53.405860 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:34:53.406094 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:53.406094 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:34:53.406525 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:53.406525 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:34:53.406525 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:34:53.406525 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:34:53.406525 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:34:53.406525 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:34:53.406525 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:53.406525 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:34:53.406525 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:34:53.406525 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:53.406525 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:34:53.406525 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:34:53.406525 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:53.406525 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:53.406525 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:34:53.406525 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:53.406820 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:34:53.406820 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:34:53.406820 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:53.406820 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:53.406820 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:53.406820 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:34:53.406820 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:53.406820 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:53.406820 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:53.406820 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:34:53.406820 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:53.406820 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:34:53.406820 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:34:53.406820 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:53.406820 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:34:53.406820 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:53.406820 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:34:53.406820 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:34:53.406820 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:53.407154 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:34:53.407154 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:53.407154 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:34:57.896863 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:34:57.897851 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:34:57.897851 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:34:57.899156 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:34:57.943268 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:34:57.943543 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:57.944424 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:34:57.944424 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:57.944424 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:34:57.944424 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:34:57.944424 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:57.944424 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:34:57.944424 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:57.944424 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:34:57.945784 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.047 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:57.945784 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:34:57.945784 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:57.945784 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:34:57.945784 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:34:57.945784 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:34:57.945784 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:34:57.945784 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:57.945784 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:57.945784 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:57.945784 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:34:57.945784 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:57.946408 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:57.946408 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:57.946408 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:34:57.946408 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:57.946408 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:34:57.946408 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:34:57.946408 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:57.946408 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:34:57.946408 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:57.946408 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:34:57.946408 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:34:57.946408 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:57.946408 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:34:57.947026 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:57.947026 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:34:57.947026 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:57.947026 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:34:57.947026 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:34:57.947026 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:34:57.947026 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:34:57.947026 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:34:57.947026 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:57.947026 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:34:57.947026 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:34:57.947026 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:57.947026 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:34:57.947026 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:34:57.947026 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:57.947802 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:57.947802 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:34:57.947802 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:57.947802 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:34:57.947802 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:34:57.947802 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:57.947802 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:57.947802 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:57.947802 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:34:57.947802 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:57.947802 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:57.947802 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:57.947802 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:34:57.947802 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:57.947802 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:34:57.947802 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:34:57.947802 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:57.947802 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:34:57.947802 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:57.950233 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:34:57.950233 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:34:57.950233 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:57.950233 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:34:57.950233 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:57.950233 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:34:58.365766 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:34:58.366564 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:34:58.366782 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:34:58.368188 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:34:58.368641 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:58.369680 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:34:58.369680 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:58.369680 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:34:58.369680 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:34:58.369680 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:58.369680 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:34:58.369680 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:58.369680 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:34:58.371482 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:58.371482 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:34:58.371482 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:58.371482 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:34:58.371482 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:34:58.371482 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:34:58.371482 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:34:58.371482 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:58.371482 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:58.371482 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:58.371482 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:34:58.371482 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:58.372744 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:58.372744 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:58.372744 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:34:58.372744 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:58.372744 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:34:58.372744 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:34:58.372744 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:58.372744 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:34:58.372744 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:58.372744 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:34:58.372744 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:34:58.372744 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:58.372744 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:34:58.373471 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:58.373471 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:34:58.373471 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:58.373471 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:34:58.373471 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:34:58.373471 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:34:58.373471 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:34:58.373471 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:34:58.373471 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:58.373471 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:34:58.373471 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:34:58.373471 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:58.373471 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:34:58.373471 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:34:58.373471 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:58.374123 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:58.374123 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:34:58.374123 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:58.374123 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:34:58.374123 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:34:58.374123 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:58.374123 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:58.374123 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:58.374123 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:34:58.374123 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:58.374123 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:58.374123 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:58.374123 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:34:58.374123 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:58.374123 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:34:58.374123 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:34:58.374123 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:58.374123 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:34:58.374123 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:58.374913 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:34:58.374913 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:34:58.374913 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:58.374913 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:34:58.374913 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:58.374913 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:34:58.381028 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:34:58.381402 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:34:58.381518 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:34:58.381669 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:34:58.382206 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:34:58.382599 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:58.383420 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:34:58.383420 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:58.383420 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:34:58.383420 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:34:58.383420 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:58.383420 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:34:58.383420 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:58.383420 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:34:58.384211 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:58.384211 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:34:58.384211 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:34:58.384211 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:34:58.384211 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:58.384211 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:34:58.384211 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:58.384211 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:34:58.384211 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:58.384211 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:34:58.384211 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:58.384211 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:34:58.384211 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:58.384211 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:34:58.384211 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:34:58.384939 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:58.384939 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:34:58.384939 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:58.384939 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:34:58.384939 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:34:58.384939 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:58.384939 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:34:58.384939 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:58.384939 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:34:58.385381 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:58.385381 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:34:58.385381 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:58.385381 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:34:58.385381 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:34:58.385381 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:34:58.385381 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:34:58.385381 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:34:58.385381 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:34:58.385381 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:34:58.385381 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:58.385381 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:58.386107 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:58.386107 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:34:58.386107 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:58.386107 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:58.386107 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:58.386107 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:34:58.386107 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:58.386107 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:34:58.386107 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:34:58.386107 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:58.386107 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:34:58.386107 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:58.386107 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:34:58.386107 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:34:58.386827 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:58.386827 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:34:58.386827 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:58.386827 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:34:58.386827 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:58.386827 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:34:58.386827 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:34:58.386827 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:34:58.386827 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:34:58.386827 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:34:58.386827 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:58.386827 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:34:58.386827 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:34:58.386827 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:58.388900 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:34:58.388900 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:34:58.388900 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:58.388900 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:58.388900 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:34:58.388900 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:58.388900 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:34:58.388900 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:34:58.388900 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:34:58.388900 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:34:58.388900 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:34:58.388900 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:58.388900 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:58.388900 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:58.388900 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:34:58.388900 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:58.388900 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:58.388900 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:58.388900 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:34:58.390699 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:58.390699 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:34:58.390699 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:34:58.390699 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:58.390699 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:34:58.390699 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:58.390699 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:34:58.390699 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:34:58.390699 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:58.390699 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:34:58.390699 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:58.390699 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:34:58.392328 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:34:58.392725 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:34:58.392813 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:34:58.392903 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:34:58.393477 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:34:58.393890 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:58.394639 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:34:58.394639 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:58.394639 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:34:58.394639 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:34:58.394639 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:58.394639 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:34:58.394639 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:58.394639 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:34:58.395464 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:58.395464 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:34:58.395464 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:34:58.395464 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:34:58.395464 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:58.395464 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:34:58.395464 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:58.395464 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:34:58.395464 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:58.395464 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:34:58.395464 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:58.395464 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:34:58.395464 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:58.395464 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:34:58.395464 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:34:58.395740 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:58.395740 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:34:58.395740 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:58.395740 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:34:58.395740 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:34:58.395740 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:58.395740 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:34:58.395740 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:58.395740 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:34:58.396364 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:58.396364 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:34:58.396364 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:58.396364 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:34:58.396364 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:34:58.396364 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:34:58.396364 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:34:58.396364 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:34:58.396364 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:34:58.396364 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:58.396364 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:58.396604 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:58.396604 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:34:58.396604 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:58.396604 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:58.396604 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:58.396604 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:34:58.396604 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:58.396604 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:34:58.396604 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:34:58.396604 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:58.396604 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:34:58.396604 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:58.396604 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:34:58.396604 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:34:58.396848 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:58.396848 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:34:58.396848 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:58.396848 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:34:58.397471 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:58.397471 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:34:58.397471 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:34:58.397471 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:34:58.397471 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:34:58.397471 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:34:58.397471 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:58.397471 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:34:58.397471 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:34:58.397471 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:58.397471 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:34:58.397471 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:34:58.397471 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:58.397471 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:58.397471 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:34:58.397471 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:58.397471 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:34:58.397795 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:34:58.397795 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:34:58.397795 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:34:58.397795 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:58.397795 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:58.397795 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:58.397795 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:34:58.397795 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:58.397795 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:58.397795 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:58.397795 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:34:58.397795 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:58.397795 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:34:58.397795 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:34:58.397795 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:58.397795 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:34:58.397795 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:58.397795 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:34:58.397795 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:34:58.398635 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:58.398635 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:34:58.398635 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:58.398635 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:34:58.402146 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:34:58.402430 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:34:58.402578 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:34:58.402694 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:34:58.403303 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:34:58.403623 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:34:58.403791 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:58.404444 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:34:58.404444 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:58.404444 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:34:58.404444 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:34:58.404444 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:58.404444 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:34:58.404444 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:58.404444 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:34:58.405247 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:58.405247 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:34:58.405247 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:58.405247 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:34:58.405247 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:34:58.405247 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:34:58.405247 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:34:58.405247 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:58.405247 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:58.405247 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:58.405247 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:34:58.405247 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:58.405619 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:34:58.405619 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:58.405619 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:34:58.405619 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:58.405619 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:34:58.405619 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:34:58.405619 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:58.405619 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:34:58.405619 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:58.405619 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:34:58.405619 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:34:58.405619 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:58.405619 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:34:58.405943 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:58.405943 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:34:58.406214 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:58.406214 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:34:58.406214 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:34:58.406214 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:34:58.406214 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:34:58.406214 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:34:58.406214 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:58.406214 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:34:58.406214 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:34:58.406214 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:58.406214 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:34:58.406214 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:34:58.406214 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:34:58.406214 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:34:58.406214 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:34:58.406214 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:34:58.407573 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:34:58.407573 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:34:58.407573 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:58.407573 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:58.407573 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:34:58.407573 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:34:58.407573 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:34:58.407573 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:34:58.407573 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:34:58.407573 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:34:58.407573 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:34:58.407573 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:34:58.407573 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:34:58.407573 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:34:58.407573 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:34:58.407573 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:34:58.407573 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:34:58.407573 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:34:58.407573 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:34:58.408677 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:34:58.408677 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:34:58.408677 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:35:02.897761 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:35:02.899228 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:35:02.900061 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:35:02.901767 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:35:02.935433 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:35:02.935999 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:02.937113 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:35:02.937113 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:02.937113 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:35:02.937113 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:35:02.937113 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:02.937113 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:35:02.937113 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:02.937113 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:35:02.938115 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.037 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:02.938115 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:35:02.938115 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:02.938115 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:35:02.938115 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:35:02.938115 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:35:02.938115 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:35:02.938115 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:02.938115 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:02.938115 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:02.938115 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:35:02.938115 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:02.938646 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:02.938646 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:02.938646 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:35:02.938646 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:02.938646 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:35:02.938646 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:35:02.938646 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:02.938646 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:35:02.938646 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:02.938646 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:35:02.938646 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:35:02.938646 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:02.938646 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:35:02.939005 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:02.939005 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:35:02.939005 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:02.939005 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:35:02.939005 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:35:02.939005 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:35:02.939005 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:35:02.939005 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:35:02.939005 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:02.939005 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:35:02.939005 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:35:02.939005 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:02.939005 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:35:02.939005 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:35:02.939005 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:02.939409 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:02.939409 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:35:02.939409 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:02.939409 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:35:02.939409 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:35:02.939409 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:02.939409 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:02.939409 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:02.939409 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:35:02.939409 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:02.939409 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:02.939409 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:02.939409 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:35:02.939409 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:02.939409 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:35:02.939409 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:35:02.939409 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:02.939409 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:35:02.939409 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:02.939884 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:35:02.939884 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:35:02.939884 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:02.939884 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:35:02.939884 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:02.939884 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:35:03.366687 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:35:03.367421 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:35:03.367624 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:35:03.369191 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:35:03.369604 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:03.370609 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:35:03.370609 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:03.370609 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:35:03.370609 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:35:03.370609 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:03.370609 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:35:03.370609 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:03.370609 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:35:03.372502 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:03.372502 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:35:03.372502 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:03.372502 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:35:03.372502 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:35:03.372502 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:35:03.372502 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:35:03.372502 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:03.372502 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:03.372502 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:03.372502 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:35:03.372502 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:03.373443 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:03.373443 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:03.373443 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:35:03.373443 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:03.373443 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:35:03.373443 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:35:03.373443 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:03.373443 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:35:03.373443 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:03.373443 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:35:03.373443 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:35:03.373443 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:03.373443 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:35:03.373863 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:03.373863 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:35:03.374472 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:03.374472 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:35:03.374472 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:35:03.374472 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:35:03.374472 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:35:03.374472 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:35:03.374472 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:03.374472 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:35:03.374472 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:35:03.374472 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:03.374472 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:35:03.374472 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:35:03.374472 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:03.374472 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:03.374472 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:35:03.374472 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:03.375698 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:35:03.375698 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:35:03.375698 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:03.375698 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:03.375698 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:03.375698 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:35:03.375698 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:03.375698 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:03.375698 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:03.375698 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:35:03.375698 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:03.375698 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:35:03.375698 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:35:03.375698 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:03.375698 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:35:03.375698 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:03.375698 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:35:03.375698 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:35:03.375698 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:03.376302 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:35:03.376302 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:03.376302 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:35:03.381643 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:35:03.382051 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:35:03.382161 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:35:03.382291 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:35:03.382943 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:35:03.383263 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:03.384230 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:35:03.384230 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:03.384230 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:35:03.384230 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:35:03.384230 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:03.384230 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:35:03.384230 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:03.384230 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:35:03.385169 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:03.385169 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:35:03.385169 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:35:03.385169 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:35:03.385169 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:03.385169 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:35:03.385169 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:03.385169 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:35:03.385169 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:03.385169 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:35:03.385169 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:03.385169 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:35:03.385169 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:03.385169 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:35:03.385169 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:35:03.385585 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:03.385585 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:35:03.385585 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:03.385585 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:35:03.385585 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:35:03.385585 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:03.385585 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:35:03.385585 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:03.385585 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:35:03.386237 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:03.386237 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:35:03.386237 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:03.386237 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:35:03.386237 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:35:03.386237 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:35:03.386237 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:35:03.386237 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:35:03.386237 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:35:03.386237 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:35:03.386237 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:03.386237 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:03.386607 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:03.386607 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:35:03.386607 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:03.386607 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:03.386607 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:03.386607 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:35:03.386607 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:03.386607 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:35:03.386607 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:35:03.386607 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:03.386607 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:35:03.386607 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:03.386607 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:35:03.386607 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:35:03.387104 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:03.387104 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:35:03.387104 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:03.387104 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:35:03.388571 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:03.388571 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:35:03.388571 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:35:03.388571 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:35:03.388571 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:35:03.388571 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:35:03.388571 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:03.388571 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:35:03.388571 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:35:03.388571 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:03.388571 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:35:03.388571 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:35:03.388571 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:03.388571 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:03.388571 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:35:03.388571 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:03.389103 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:35:03.389103 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:35:03.389103 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:35:03.389103 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:35:03.389103 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:35:03.389103 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:03.389103 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:03.389103 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:03.389103 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:35:03.389103 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:03.389103 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:03.389103 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:03.389103 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:35:03.389103 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:03.389103 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:35:03.389103 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:35:03.389103 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:03.389103 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:35:03.389626 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:03.389626 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:35:03.389626 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:35:03.389626 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:03.389626 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:35:03.389626 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:03.389626 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:35:03.392795 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:35:03.393142 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:35:03.393285 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:35:03.393513 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:35:03.394081 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:35:03.394514 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:03.395455 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:35:03.395455 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:03.395455 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:35:03.395455 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:35:03.395455 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:03.395455 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:35:03.395455 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:03.395455 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:35:03.396995 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:03.396995 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:35:03.396995 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:35:03.396995 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:35:03.396995 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:03.396995 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:35:03.396995 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:03.396995 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:35:03.396995 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:03.396995 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:35:03.396995 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:03.396995 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:35:03.396995 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:03.396995 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:35:03.396995 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:35:03.397623 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:03.397623 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:35:03.397623 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:03.397623 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:35:03.397623 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:35:03.397623 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:03.397623 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:35:03.397623 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:03.397623 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:35:03.398887 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:03.398887 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:35:03.398887 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:03.398887 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:35:03.398887 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:35:03.398887 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:35:03.398887 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:35:03.398887 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:35:03.398887 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:35:03.398887 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:03.398887 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:03.399435 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:03.399435 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:35:03.399435 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:03.399435 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:03.399435 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:03.399435 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:35:03.399435 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:03.399435 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:35:03.399435 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:35:03.399435 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:03.399435 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:35:03.399435 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:03.399435 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:35:03.399435 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:35:03.400045 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:03.400045 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:35:03.400045 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:03.400045 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:35:03.401147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:03.401147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:35:03.401147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:35:03.401147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:35:03.401147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:35:03.401147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:35:03.401147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:03.401147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:35:03.401147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:35:03.401147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:03.401147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:35:03.401147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:35:03.401147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:03.401147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:03.401147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:35:03.401147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:03.401147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:35:03.401816 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:35:03.401816 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:35:03.401816 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:35:03.401816 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:03.401816 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:03.401816 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:03.401816 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:35:03.401816 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:03.401816 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:03.401816 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:03.401816 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:35:03.401816 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:03.401816 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:35:03.401816 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:35:03.401816 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:03.401816 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:35:03.401816 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:03.401816 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:35:03.401816 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:35:03.402332 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:03.402332 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:35:03.402332 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:03.402332 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:35:03.402608 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:35:03.403099 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:35:03.403312 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:35:03.403639 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:35:03.404182 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:35:03.404599 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:35:03.404854 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:03.405679 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:35:03.405679 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:03.405679 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:35:03.405679 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:35:03.405679 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:03.405679 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:35:03.405679 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:03.405679 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:35:03.406621 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:03.406621 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:35:03.406621 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:03.406621 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:35:03.406621 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:35:03.406621 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:35:03.406621 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:35:03.406621 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:03.406621 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:03.406621 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:03.406621 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:35:03.406621 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:03.406981 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:03.406981 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:03.406981 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:35:03.406981 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:03.406981 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:35:03.406981 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:35:03.406981 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:03.406981 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:35:03.406981 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:03.406981 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:35:03.406981 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:35:03.406981 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:03.406981 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:35:03.407347 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:03.407347 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:35:03.407647 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:03.407647 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:35:03.407647 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:35:03.407647 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:35:03.407647 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:35:03.407647 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:35:03.407647 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:03.407647 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:35:03.407647 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:35:03.407647 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:03.407647 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:35:03.407647 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:35:03.407647 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:03.407647 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:03.407647 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:35:03.407647 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:03.408118 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:35:03.408118 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:35:03.408118 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:03.408118 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:03.408118 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:03.408118 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:35:03.408118 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:03.408118 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:03.408118 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:03.408118 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:35:03.408118 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:03.408118 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:35:03.408118 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:35:03.408118 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:03.408118 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:35:03.408118 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:03.408118 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:35:03.408118 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:35:03.408118 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:03.408631 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:35:03.408631 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:03.408631 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:35:07.899402 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:35:07.900183 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:35:07.901315 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:35:07.901812 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:35:07.939187 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:35:07.939187 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:07.939782 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:35:07.939782 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:07.939782 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:35:07.939782 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:35:07.939782 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:07.939782 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:35:07.939782 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:07.939782 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:35:07.940731 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.039 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:07.940731 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:35:07.940731 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:07.940731 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:35:07.940731 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:35:07.940731 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:35:07.940731 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:35:07.940731 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:07.940731 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:07.940731 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:07.940731 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:35:07.940731 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:07.941597 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:07.941597 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:07.941597 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:35:07.941597 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:07.941597 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:35:07.941597 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:35:07.941597 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:07.941597 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:35:07.941597 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:07.941597 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:35:07.941597 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:35:07.941597 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:07.941597 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:35:07.942616 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:07.942616 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:35:07.942616 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:07.942616 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:35:07.942616 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:35:07.942616 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:35:07.942616 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:35:07.942616 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:35:07.942616 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:07.942616 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:35:07.942616 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:35:07.942616 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:07.942616 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:35:07.942616 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:35:07.942616 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:07.943649 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:07.943649 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:35:07.943649 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:07.943649 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:35:07.943649 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:35:07.943649 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:07.943649 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:07.943649 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:07.943649 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:35:07.943649 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:07.943649 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:07.943649 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:07.943649 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:35:07.943649 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:07.943649 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:35:07.943649 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:35:07.943649 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:07.943649 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:35:07.943649 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:07.944390 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:35:07.944390 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:35:07.944390 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:07.944390 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:35:07.944390 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:07.944390 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:35:08.367550 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:35:08.368228 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:35:08.368413 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:35:08.369125 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:35:08.369297 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:08.370072 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:35:08.370072 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:08.370072 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:35:08.370072 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:35:08.370072 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:08.370072 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:35:08.370072 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:08.370072 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:35:08.371598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:08.371598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:35:08.371598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:08.371598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:35:08.371598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:35:08.371598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:35:08.371598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:35:08.371598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:08.371598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:08.371598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:08.371598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:35:08.371598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:08.372216 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:08.372216 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:08.372216 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:35:08.372216 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:08.372216 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:35:08.372216 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:35:08.372216 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:08.372216 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:35:08.372216 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:08.372216 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:35:08.372216 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:35:08.372216 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:08.372216 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:35:08.372847 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:08.372847 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:35:08.373285 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:08.373285 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:35:08.373285 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:35:08.373285 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:35:08.373285 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:35:08.373285 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:35:08.373285 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:08.373285 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:35:08.373285 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:35:08.373285 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:08.373285 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:35:08.373285 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:35:08.373285 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:08.373285 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:08.373285 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:35:08.373285 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:08.374172 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:35:08.374172 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:35:08.374172 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:08.374172 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:08.374172 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:08.374172 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:35:08.374172 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:08.374172 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:08.374172 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:08.374172 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:35:08.374172 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:08.374172 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:35:08.374172 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:35:08.374172 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:08.374172 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:35:08.374172 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:08.374172 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:35:08.374172 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:35:08.374172 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:08.375898 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:35:08.375898 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:08.375898 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:35:08.382423 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:35:08.382795 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:35:08.382930 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:35:08.383084 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:35:08.383387 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:35:08.383575 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:08.384012 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:35:08.384012 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:08.384012 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:35:08.384012 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:35:08.384012 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:08.384012 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:35:08.384012 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:08.384012 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:35:08.384900 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:08.384900 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:35:08.384900 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:35:08.384900 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:35:08.384900 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:08.384900 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:35:08.384900 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:08.384900 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:35:08.384900 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:08.384900 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:35:08.384900 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:08.384900 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:35:08.384900 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:08.384900 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:35:08.384900 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:35:08.385469 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:08.385469 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:35:08.385469 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:08.385469 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:35:08.385469 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:35:08.385469 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:08.385469 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:35:08.385469 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:08.385469 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:35:08.386039 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:08.386039 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:35:08.386039 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:08.386039 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:35:08.386039 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:35:08.386039 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:35:08.386039 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:35:08.386039 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:35:08.386039 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:35:08.386039 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:35:08.386039 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:08.386039 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:08.386473 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:08.386473 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:35:08.386473 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:08.386473 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:08.386473 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:08.386473 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:35:08.386473 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:08.386473 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:35:08.386473 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:35:08.386473 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:08.386473 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:35:08.386473 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:08.386473 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:35:08.386473 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:35:08.386896 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:08.386896 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:35:08.386896 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:08.386896 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:35:08.387570 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:08.387570 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:35:08.387570 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:35:08.387570 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:35:08.387570 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:35:08.387570 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:35:08.387570 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:08.387570 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:35:08.387570 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:35:08.387570 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:08.387570 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:35:08.387570 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:35:08.387570 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:08.387570 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:08.387570 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:35:08.387570 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:08.388149 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:35:08.388149 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:35:08.388149 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:35:08.388149 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:35:08.388149 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:35:08.388149 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:08.388149 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:08.388149 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:08.388149 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:35:08.388149 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:08.388149 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:08.388149 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:08.388149 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:35:08.388149 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:08.388149 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:35:08.388149 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:35:08.388149 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:08.388149 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:35:08.388675 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:08.388675 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:35:08.388675 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:35:08.388675 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:08.388675 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:35:08.388675 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:08.388675 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:35:08.393684 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:35:08.393684 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:35:08.393817 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:35:08.394010 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:35:08.394275 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:35:08.394529 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:08.394916 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:35:08.394916 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:08.394916 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:35:08.394916 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:35:08.394916 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:08.394916 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:35:08.394916 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:08.394916 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:35:08.395748 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:08.395748 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:35:08.395748 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:35:08.395748 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:35:08.395748 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:08.395748 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:35:08.395748 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:08.395748 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:35:08.395748 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:08.395748 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:35:08.395748 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:08.395748 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:35:08.395748 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:08.395748 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:35:08.395748 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:35:08.396199 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:08.396199 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:35:08.396199 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:08.396199 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:35:08.396199 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:35:08.396199 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:08.396199 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:35:08.396199 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:08.396199 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:35:08.396863 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:08.396863 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:35:08.396863 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:08.396863 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:35:08.396863 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:35:08.396863 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:35:08.396863 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:35:08.396863 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:35:08.396863 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:35:08.396863 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:08.396863 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:08.397234 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:08.397234 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:35:08.397234 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:08.397234 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:08.397234 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:08.397234 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:35:08.397234 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:08.397234 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:35:08.397234 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:35:08.397234 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:08.397234 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:35:08.397234 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:08.397234 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:35:08.397234 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:35:08.397704 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:08.397704 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:35:08.397704 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:08.397704 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:35:08.398260 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:08.398260 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:35:08.398260 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:35:08.398260 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:35:08.398260 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:35:08.398260 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:35:08.398260 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:08.398260 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:35:08.398260 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:35:08.398260 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:08.398260 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:35:08.398260 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:35:08.398260 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:08.398260 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:08.398260 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:35:08.398260 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:08.398260 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:35:08.398852 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:35:08.398852 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:35:08.398852 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:35:08.398852 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:08.398852 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:08.398852 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:08.398852 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:35:08.398852 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:08.398852 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:08.398852 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:08.398852 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:35:08.398852 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:08.398852 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:35:08.398852 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:35:08.398852 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:08.398852 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:35:08.398852 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:08.398852 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:35:08.398852 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:35:08.399396 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:08.399396 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:35:08.399396 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:08.399396 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:35:08.403139 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:35:08.403487 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:35:08.403636 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:35:08.403831 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:35:08.404069 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:35:08.404247 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:35:08.404370 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:08.404782 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:35:08.404782 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:08.404782 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:35:08.404782 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:35:08.404782 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:08.404782 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:35:08.404782 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:08.404782 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:35:08.405718 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:08.405718 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:35:08.405718 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:08.405718 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:35:08.405718 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:35:08.405718 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:35:08.405718 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:35:08.405718 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:08.405718 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:08.405718 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:08.405718 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:35:08.405718 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:08.406099 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:08.406099 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:08.406099 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:35:08.406099 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:08.406099 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:35:08.406099 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:35:08.406099 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:08.406099 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:35:08.406099 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:08.406099 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:35:08.406099 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:35:08.406099 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:08.406099 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:35:08.406528 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:08.406528 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:35:08.407036 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:08.407036 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:35:08.407036 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:35:08.407036 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:35:08.407036 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:35:08.407036 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:35:08.407036 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:08.407036 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:35:08.407036 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:35:08.407036 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:08.407036 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:35:08.407036 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:35:08.407036 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:08.407036 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:08.407036 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:35:08.407036 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:08.407532 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:35:08.407532 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:35:08.407532 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:08.407532 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:08.407532 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:08.407532 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:35:08.407532 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:08.407532 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:08.407532 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:08.407532 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:35:08.407532 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:08.407532 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:35:08.407532 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:35:08.407532 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:08.407532 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:35:08.407532 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:08.407532 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:35:08.407532 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:35:08.407532 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:08.408103 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:35:08.408103 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:08.408103 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:35:12.900148 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:35:12.902019 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:35:12.902019 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:35:12.902748 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:35:12.938230 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:35:12.938230 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:12.938503 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:35:12.938503 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:12.938503 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:35:12.938503 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:35:12.938503 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:12.938503 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:35:12.938503 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:12.938503 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:35:12.939666 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.037 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:12.939666 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:35:12.939666 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:12.939666 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:35:12.939666 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:35:12.939666 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:35:12.939666 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:35:12.939666 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:12.939666 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:12.939666 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:12.939666 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:35:12.939666 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:12.940069 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:12.940069 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:12.940069 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:35:12.940069 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:12.940069 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:35:12.940069 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:35:12.940069 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:12.940069 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:35:12.940069 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:12.940069 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:35:12.940069 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:35:12.940069 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:12.940069 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:35:12.940516 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:12.940516 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:35:12.940846 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:12.940846 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:35:12.940846 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:35:12.940846 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:35:12.940846 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:35:12.940846 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:35:12.940846 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:12.940846 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:35:12.940846 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:35:12.940846 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:12.940846 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:35:12.940846 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:35:12.940846 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:12.940846 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:12.940846 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:35:12.940846 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:12.941372 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:35:12.941372 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:35:12.941372 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:12.941372 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:12.941372 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:12.941372 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:35:12.941372 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:12.941372 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:12.941372 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:12.941372 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:35:12.941372 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:12.941372 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:35:12.941372 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:35:12.941372 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:12.941372 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:35:12.941372 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:12.941372 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:35:12.941372 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:35:12.941372 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:12.941943 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:35:12.941943 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:12.941943 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:35:13.368366 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:35:13.369128 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:35:13.369340 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:35:13.371528 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:35:13.371528 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:13.371877 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:35:13.371877 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:13.371877 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:35:13.371877 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:35:13.371877 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:13.371877 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:35:13.371877 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:13.371877 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:35:13.373619 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:13.373619 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:35:13.373619 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:13.373619 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:35:13.373619 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:35:13.373619 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:35:13.373619 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:35:13.373619 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:13.373619 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:13.373619 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:13.373619 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:35:13.373619 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:13.374185 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:13.374185 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:13.374185 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:35:13.374185 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:13.374185 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:35:13.374185 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:35:13.374185 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:13.374185 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:35:13.374185 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:13.374185 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:35:13.374185 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:35:13.374185 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:13.374185 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:35:13.374931 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:13.374931 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:35:13.375286 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:13.375286 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:35:13.375286 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:35:13.375286 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:35:13.375286 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:35:13.375286 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:35:13.375286 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:13.375286 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:35:13.375286 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:35:13.375286 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:13.375286 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:35:13.375286 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:35:13.375286 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:13.375286 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:13.375286 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:35:13.375286 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:13.376091 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:35:13.376091 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:35:13.376091 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:13.376091 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:13.376091 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:13.376091 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:35:13.376091 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:13.376091 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:13.376091 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:13.376091 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:35:13.376091 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:13.376091 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:35:13.376091 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:35:13.376091 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:13.376091 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:35:13.376091 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:13.376091 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:35:13.376091 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:35:13.376091 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:13.376946 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:35:13.376946 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:13.376946 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:35:13.383075 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:35:13.383467 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:35:13.383590 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:35:13.383661 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:35:13.384122 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:35:13.384388 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:13.384957 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:35:13.384957 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:13.384957 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:35:13.384957 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:35:13.384957 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:13.384957 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:35:13.384957 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:13.384957 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:35:13.385878 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:13.385878 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:35:13.385878 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:35:13.385878 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:35:13.385878 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:13.385878 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:35:13.385878 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:13.385878 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:35:13.385878 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:13.385878 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:35:13.385878 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:13.385878 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:35:13.385878 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:13.385878 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:35:13.385878 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:35:13.386582 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:13.386582 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:35:13.386582 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:13.386582 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:35:13.386582 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:35:13.386582 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:13.386582 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:35:13.386582 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:13.386582 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:35:13.386979 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:13.386979 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:35:13.386979 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:13.386979 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:35:13.386979 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:35:13.386979 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:35:13.386979 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:35:13.386979 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:35:13.386979 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:35:13.386979 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:35:13.386979 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:13.386979 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:13.387579 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:13.387579 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:35:13.387579 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:13.387579 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:13.387579 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:13.387579 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:35:13.387579 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:13.387579 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:35:13.387579 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:35:13.387579 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:13.387579 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:35:13.387579 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:13.387579 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:35:13.387579 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:35:13.388183 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:13.388183 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:35:13.388183 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:13.388183 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:35:13.388183 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:13.388183 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:35:13.388183 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:35:13.388183 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:35:13.388183 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:35:13.388183 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:35:13.388183 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:13.388183 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:35:13.388183 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:35:13.388183 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:13.388845 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:35:13.388845 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:35:13.388845 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:13.388845 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:13.388845 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:35:13.388845 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:13.388845 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:35:13.388845 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:35:13.388845 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:35:13.388845 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:35:13.388845 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:35:13.388845 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:13.388845 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:13.388845 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:13.388845 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:35:13.388845 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:13.388845 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:13.388845 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:13.388845 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:35:13.389725 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:13.389725 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:35:13.389725 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:35:13.389725 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:13.389725 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:35:13.389725 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:13.389725 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:35:13.389725 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:35:13.389725 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:13.389725 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:35:13.389725 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:13.389725 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:35:13.393951 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:35:13.394499 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:35:13.394663 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:35:13.394854 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:35:13.395261 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:35:13.395581 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:13.396195 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:35:13.396195 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:13.396195 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:35:13.396195 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:35:13.396195 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:13.396195 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:35:13.396195 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:13.396195 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:35:13.397074 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:13.397074 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:35:13.397074 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:35:13.397074 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:35:13.397074 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:13.397074 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:35:13.397074 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:13.397074 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:35:13.397074 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:13.397074 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:35:13.397074 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:13.397074 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:35:13.397074 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:13.397074 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:35:13.397074 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:35:13.397532 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:13.397532 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:35:13.397532 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:13.397532 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:35:13.397532 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:35:13.397532 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:13.397532 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:35:13.397532 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:13.397532 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:35:13.398336 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:13.398336 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:35:13.398336 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:13.398336 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:35:13.398336 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:35:13.398336 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:35:13.398336 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:35:13.398336 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:35:13.398336 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:35:13.398336 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:13.398336 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:13.398708 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:13.398708 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:35:13.398708 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:13.398708 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:13.398708 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:13.398708 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:35:13.398708 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:13.398708 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:35:13.398708 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:35:13.398708 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:13.398708 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:35:13.398708 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:13.398708 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:35:13.398708 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:35:13.399080 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:13.399080 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:35:13.399080 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:13.399080 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:35:13.400262 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:13.400262 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:35:13.400262 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:35:13.400262 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:35:13.400262 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:35:13.400262 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:35:13.400262 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:13.400262 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:35:13.400262 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:35:13.400262 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:13.400262 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:35:13.400262 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:35:13.400262 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:13.400262 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:13.400262 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:35:13.400262 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:13.400262 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:35:13.400762 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:35:13.400762 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:35:13.400762 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:35:13.400762 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:13.400762 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:13.400762 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:13.400762 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:35:13.400762 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:13.400762 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:13.400762 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:13.400762 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:35:13.400762 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:13.400762 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:35:13.400762 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:35:13.400762 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:13.400762 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:35:13.400762 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:13.400762 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:35:13.400762 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:35:13.401277 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:13.401277 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:35:13.401277 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:13.401277 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:35:13.403636 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:35:13.404075 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:35:13.404197 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:35:13.404412 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:35:13.404762 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:35:13.404986 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:35:13.405115 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:13.405759 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:35:13.405759 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:13.405759 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:35:13.405759 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:35:13.405759 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:13.405759 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:35:13.405759 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:13.405759 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:35:13.406726 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:13.406726 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:35:13.406726 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:13.406726 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:35:13.406726 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:35:13.406726 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:35:13.406726 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:35:13.406726 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:13.406726 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:13.406726 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:13.406726 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:35:13.406726 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:13.407059 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:13.407059 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:13.407059 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:35:13.407059 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:13.407059 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:35:13.407059 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:35:13.407059 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:13.407059 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:35:13.407059 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:13.407059 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:35:13.407059 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:35:13.407059 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:13.407059 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:35:13.407421 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:13.407421 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:35:13.407803 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:13.407803 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:35:13.407803 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:35:13.407803 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:35:13.407803 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:35:13.407803 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:35:13.407803 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:13.407803 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:35:13.407803 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:35:13.407803 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:13.407803 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:35:13.407803 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:35:13.407803 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:13.407803 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:13.407803 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:35:13.407803 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:13.408258 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:35:13.408258 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:35:13.408258 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:13.408258 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:13.408258 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:13.408258 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:35:13.408258 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:13.408258 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:13.408258 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:13.408258 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:35:13.408258 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:13.408258 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:35:13.408258 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:35:13.408258 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:13.408258 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:35:13.408258 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:13.408258 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:35:13.408258 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:35:13.408258 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:13.408789 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:35:13.408789 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:13.408789 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:35:17.900999 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:35:17.901904 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:35:17.901904 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:35:17.903016 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:35:17.949168 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:35:17.949667 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:17.950900 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:35:17.950900 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:17.950900 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:35:17.950900 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:35:17.950900 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:17.950900 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:35:17.950900 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:17.950900 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:35:17.951855 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.049 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:17.951855 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:35:17.951855 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:17.951855 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:35:17.951855 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:35:17.951855 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:35:17.951855 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:35:17.951855 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:17.951855 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:17.951855 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:17.951855 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:35:17.951855 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:17.953112 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:17.953112 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:17.953112 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:35:17.953112 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:17.953112 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:35:17.953112 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:35:17.953112 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:17.953112 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:35:17.953112 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:17.953112 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:35:17.953112 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:35:17.953112 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:17.953112 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:35:17.954288 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:17.954288 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:35:17.954288 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:17.954288 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:35:17.954288 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:35:17.954288 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:35:17.954288 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:35:17.954288 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:35:17.954288 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:17.954288 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:35:17.954288 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:35:17.954288 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:17.954288 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:35:17.954288 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:35:17.954288 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:17.954980 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:17.954980 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:35:17.954980 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:17.954980 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:35:17.954980 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:35:17.954980 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:17.954980 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:17.954980 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:17.954980 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:35:17.954980 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:17.954980 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:17.954980 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:17.954980 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:35:17.954980 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:17.954980 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:35:17.954980 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:35:17.954980 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:17.954980 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:35:17.954980 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:17.956891 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:35:17.956891 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:35:17.956891 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:17.956891 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:35:17.956891 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:17.956891 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:35:18.369364 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:35:18.370182 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:35:18.370413 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:35:18.371603 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:35:18.372010 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:18.373082 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:35:18.373082 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:18.373082 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:35:18.373082 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:35:18.373082 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:18.373082 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:35:18.373082 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:18.373082 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:35:18.374958 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:18.374958 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:35:18.374958 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:18.374958 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:35:18.374958 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:35:18.374958 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:35:18.374958 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:35:18.374958 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:18.374958 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:18.374958 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:18.374958 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:35:18.374958 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:18.375602 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:18.375602 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:18.375602 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:35:18.375602 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:18.375602 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:35:18.375602 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:35:18.375602 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:18.375602 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:35:18.375602 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:18.375602 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:35:18.375602 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:35:18.375602 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:18.375602 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:35:18.376239 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:18.376239 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:35:18.376692 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:18.376692 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:35:18.376692 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:35:18.376692 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:35:18.376692 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:35:18.376692 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:35:18.376692 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:18.376692 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:35:18.376692 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:35:18.376692 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:18.376692 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:35:18.376692 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:35:18.376692 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:18.376692 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:18.376692 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:35:18.376692 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:18.377493 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:35:18.377493 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:35:18.377493 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:18.377493 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:18.377493 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:18.377493 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:35:18.377493 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:18.377493 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:18.377493 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:18.377493 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:35:18.377493 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:18.377493 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:35:18.377493 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:35:18.377493 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:18.377493 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:35:18.377493 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:18.377493 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:35:18.377493 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:35:18.377493 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:18.378385 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:35:18.378385 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:18.378385 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:35:18.383564 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:35:18.383885 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:35:18.384004 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:35:18.384127 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:35:18.384741 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:35:18.385057 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:18.385877 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:35:18.385877 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:18.385877 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:35:18.385877 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:35:18.385877 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:18.385877 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:35:18.385877 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:18.385877 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:35:18.386684 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:18.386684 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:35:18.386684 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:35:18.386684 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:35:18.386684 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:18.386684 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:35:18.386684 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:18.386684 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:35:18.386684 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:18.386684 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:35:18.386684 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:18.386684 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:35:18.386684 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:18.386684 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:35:18.386684 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:35:18.387215 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:18.387215 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:35:18.387215 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:18.387215 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:35:18.387215 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:35:18.387215 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:18.387215 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:35:18.387215 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:18.387215 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:35:18.387534 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:18.387534 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:35:18.387534 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:18.387534 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:35:18.387534 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:35:18.387534 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:35:18.387534 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:35:18.387534 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:35:18.387534 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:35:18.387534 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:35:18.387534 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:18.387534 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:18.387964 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:18.387964 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:35:18.387964 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:18.387964 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:18.387964 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:18.387964 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:35:18.387964 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:18.387964 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:35:18.387964 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:35:18.387964 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:18.387964 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:35:18.387964 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:18.387964 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:35:18.387964 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:35:18.388428 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:18.388428 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:35:18.388428 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:18.388428 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:35:18.388564 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:18.388564 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:35:18.388564 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:35:18.388564 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:35:18.388564 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:35:18.388564 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:35:18.388564 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:18.388564 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:35:18.388564 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:35:18.388564 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:18.388564 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:35:18.388564 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:35:18.388564 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:18.388564 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:18.388564 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:35:18.388564 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:18.389190 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:35:18.389190 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:35:18.389190 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:35:18.389190 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:35:18.389190 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:35:18.389190 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:18.389190 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:18.389190 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:18.389190 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:35:18.389190 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:18.389190 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:18.389190 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:18.389190 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:35:18.389190 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:18.389190 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:35:18.389190 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:35:18.389190 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:18.389190 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:35:18.389839 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:18.389839 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:35:18.389839 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:35:18.389839 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:18.389839 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:35:18.389839 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:18.389839 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:35:18.394452 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:35:18.394777 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:35:18.394885 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:35:18.395046 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:35:18.395568 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:35:18.395929 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:18.396655 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:35:18.396655 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:18.396655 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:35:18.396655 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:35:18.396655 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:18.396655 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:35:18.396655 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:18.396655 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:35:18.397417 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:18.397417 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:35:18.397417 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:35:18.397417 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:35:18.397417 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:18.397417 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:35:18.397417 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:18.397417 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:35:18.397417 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:18.397417 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:35:18.397417 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:18.397417 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:35:18.397417 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:18.397417 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:35:18.397417 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:35:18.397888 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:18.397888 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:35:18.397888 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:18.397888 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:35:18.397888 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:35:18.397888 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:18.397888 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:35:18.397888 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:18.397888 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:35:18.398438 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:18.398438 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:35:18.398438 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:18.398438 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:35:18.398438 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:35:18.398438 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:35:18.398438 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:35:18.398438 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:35:18.398438 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:35:18.398438 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:18.398438 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:18.398796 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:18.398796 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:35:18.398796 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:18.398796 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:18.398796 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:18.398796 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:35:18.398796 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:18.398796 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:35:18.398796 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:35:18.398796 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:18.398796 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:35:18.398796 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:18.398796 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:35:18.398796 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:35:18.399195 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:18.399195 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:35:18.399195 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:18.399195 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:35:18.399643 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:18.399643 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:35:18.399643 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:35:18.399643 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:35:18.399643 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:35:18.399643 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:35:18.399643 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:18.399643 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:35:18.399643 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:35:18.399643 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:18.399643 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:35:18.399643 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:35:18.399643 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:18.399643 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:18.399643 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:35:18.399643 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:18.399643 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:35:18.400165 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:35:18.400165 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:35:18.400165 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:35:18.400165 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:18.400165 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:18.400165 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:18.400165 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:35:18.400165 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:18.400165 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:18.400165 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:18.400165 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:35:18.400165 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:18.400165 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:35:18.400165 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:35:18.400165 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:18.400165 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:35:18.400165 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:18.400165 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:35:18.400165 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:35:18.401012 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:18.401012 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:35:18.401012 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:18.401012 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:35:18.404246 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:35:18.404634 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:35:18.404756 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:35:18.404904 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:35:18.405399 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:35:18.405781 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:35:18.405975 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:18.406709 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:35:18.406709 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:18.406709 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:35:18.406709 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:35:18.406709 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:18.406709 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:35:18.406709 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:18.406709 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:35:18.407739 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:18.407739 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:35:18.407739 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:18.407739 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:35:18.407739 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:35:18.407739 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:35:18.407739 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:35:18.407739 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:18.407739 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:18.407739 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:18.407739 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:35:18.407739 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:18.408363 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:18.408363 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:18.408363 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:35:18.408363 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:18.408363 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:35:18.408363 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:35:18.408363 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:18.408363 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:35:18.408363 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:18.408363 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:35:18.408363 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:35:18.408363 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:18.408363 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:35:18.408752 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:18.408752 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:35:18.408752 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:18.408752 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:35:18.408752 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:35:18.408752 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:35:18.408752 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:35:18.408752 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:35:18.408752 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:18.408752 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:35:18.408752 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:35:18.408752 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:18.408752 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:35:18.408752 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:35:18.408752 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:18.409224 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:18.409224 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:35:18.409224 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:18.409224 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:35:18.409224 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:35:18.409224 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:18.409224 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:18.409224 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:18.409224 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:35:18.409224 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:18.409224 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:18.409224 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:18.409224 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:35:18.409224 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:18.409224 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:35:18.409224 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:35:18.409224 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:18.409224 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:35:18.409224 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:18.409848 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:35:18.409848 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:35:18.409848 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:18.409848 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:35:18.409848 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:18.409848 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:35:22.902629 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:35:22.903613 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:35:22.903898 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:35:22.905633 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:35:23.314334 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:35:23.316202 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:23.316202 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:35:23.316202 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:23.316202 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:35:23.316202 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:35:23.316202 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:23.316202 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:35:23.316202 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:23.316202 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:35:23.317729 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.412 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:23.317729 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:35:23.317729 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:23.317729 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:35:23.317729 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:35:23.317729 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:35:23.317729 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:35:23.317729 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:23.317729 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:23.317729 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:23.317729 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:35:23.317729 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:23.318821 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:23.318821 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:23.318821 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:35:23.318821 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:23.318821 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:35:23.318821 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:35:23.318821 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:23.318821 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:35:23.318821 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:23.318821 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:35:23.318821 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:35:23.318821 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:23.318821 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:35:23.320244 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:23.320244 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:35:23.320244 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:23.320244 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:35:23.320244 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:35:23.320244 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:35:23.320244 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:35:23.320244 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:35:23.320244 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:23.320244 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:35:23.320244 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:35:23.320244 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:23.320244 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:35:23.320244 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:35:23.320244 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:23.323597 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:23.323597 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:35:23.323597 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:23.323597 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:35:23.323597 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:35:23.323597 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:23.323597 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:23.323597 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:23.323597 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:35:23.323597 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:23.323597 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:23.323597 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:23.323597 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:35:23.323597 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:23.323597 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:35:23.323597 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:35:23.323597 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:23.323597 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:35:23.323597 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:23.324813 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:35:23.324813 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:35:23.324813 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:23.324813 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:35:23.324813 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:23.324813 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:35:23.370620 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:35:23.371229 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:35:23.371518 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:35:23.373337 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:35:23.373337 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:23.374391 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:35:23.374391 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:23.374391 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:35:23.374391 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:35:23.374391 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:23.374391 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:35:23.374391 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:23.374391 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:35:23.376017 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:23.376017 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:35:23.376017 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:23.376017 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:35:23.376017 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:35:23.376017 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:35:23.376017 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:35:23.376017 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:23.376017 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:23.376017 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:23.376017 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:35:23.376017 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:23.376530 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:23.376530 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:23.376530 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:35:23.376530 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:23.376530 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:35:23.376530 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:35:23.376530 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:23.376530 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:35:23.376530 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:23.376530 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:35:23.376530 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:35:23.376530 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:23.376530 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:35:23.376875 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:23.376875 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:35:23.377889 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:23.377889 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:35:23.377889 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:35:23.377889 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:35:23.377889 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:35:23.377889 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:35:23.377889 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:23.377889 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:35:23.377889 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:35:23.377889 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:23.377889 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:35:23.377889 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:35:23.377889 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:23.377889 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:23.377889 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:35:23.377889 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:23.378433 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:35:23.378433 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:35:23.378433 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:23.378433 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:23.378433 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:23.378433 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:35:23.378433 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:23.378433 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:23.378433 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:23.378433 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:35:23.378433 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:23.378433 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:35:23.378433 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:35:23.378433 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:23.378433 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:35:23.378433 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:23.378433 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:35:23.378433 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:35:23.378433 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:23.378925 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:35:23.378925 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:23.378925 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:35:23.384105 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:35:23.384311 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:35:23.384455 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:35:23.384576 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:35:23.385178 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:35:23.385638 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:23.386623 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:35:23.386623 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:23.386623 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:35:23.386623 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:35:23.386623 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:23.386623 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:35:23.386623 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:23.386623 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:35:23.387376 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:23.387376 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:35:23.387376 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:35:23.387376 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:35:23.387376 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:23.387376 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:35:23.387376 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:23.387376 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:35:23.387376 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:23.387376 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:35:23.387376 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:23.387376 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:35:23.387376 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:23.387376 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:35:23.387376 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:35:23.387846 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:23.387846 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:35:23.387846 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:23.387846 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:35:23.387846 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:35:23.387846 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:23.387846 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:35:23.387846 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:23.387846 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:35:23.388378 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:23.388378 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:35:23.388378 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:23.388378 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:35:23.388378 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:35:23.388378 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:35:23.388378 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:35:23.388378 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:35:23.388378 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:35:23.388378 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:35:23.388378 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:23.388378 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:23.389349 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:23.389349 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:35:23.389349 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:23.389349 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:23.389349 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:23.389349 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:35:23.389349 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:23.389349 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:35:23.389349 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:35:23.389349 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:23.389349 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:35:23.389349 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:23.389349 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:35:23.389349 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:35:23.389987 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:23.389987 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:35:23.389987 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:23.389987 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:35:23.389987 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:23.389987 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:35:23.389987 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:35:23.389987 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:35:23.389987 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:35:23.389987 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:35:23.389987 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:23.389987 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:35:23.389987 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:35:23.389987 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:23.390350 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:35:23.390350 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:35:23.390350 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:23.390350 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:23.390350 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:35:23.390350 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:23.390350 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:35:23.390350 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:35:23.390350 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:35:23.390350 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:35:23.390350 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:35:23.390350 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:23.390350 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:23.390350 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:23.390350 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:35:23.390350 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:23.390350 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:23.390350 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:23.390350 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:35:23.390903 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:23.390903 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:35:23.390903 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:35:23.390903 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:23.390903 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:35:23.390903 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:23.390903 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:35:23.390903 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:35:23.390903 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:23.390903 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:35:23.390903 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:23.390903 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:35:23.395060 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:35:23.395414 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:35:23.395584 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:35:23.395784 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:35:23.396457 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:35:23.396814 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:23.397496 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:35:23.397496 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:23.397496 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:35:23.397496 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:35:23.397496 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:23.397496 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:35:23.397496 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:23.397496 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:35:23.398261 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:23.398261 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:35:23.398261 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:35:23.398261 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:35:23.398261 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:23.398261 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:35:23.398261 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:23.398261 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:35:23.398261 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:23.398261 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:35:23.398261 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:23.398261 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:35:23.398261 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:23.398261 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:35:23.398261 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:35:23.398707 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:23.398707 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:35:23.398707 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:23.398707 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:35:23.398707 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:35:23.398707 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:23.398707 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:35:23.398707 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:23.398707 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:35:23.399245 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:23.399245 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:35:23.399245 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:23.399245 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:35:23.399245 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:35:23.399245 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:35:23.399245 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:35:23.399245 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:35:23.399245 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:35:23.399245 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:23.399245 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:23.399996 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:23.399996 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:35:23.399996 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:23.399996 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:23.399996 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:23.399996 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:35:23.399996 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:23.399996 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:35:23.399996 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:35:23.399996 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:23.399996 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:35:23.399996 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:23.399996 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:35:23.399996 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:35:23.400672 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:23.400672 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:35:23.400672 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:23.400672 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:35:23.400672 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:23.400672 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:35:23.400672 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:35:23.400672 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:35:23.400672 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:35:23.400672 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:35:23.400672 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:23.400672 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:35:23.400672 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:35:23.400672 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:23.401026 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:35:23.401026 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:35:23.401026 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:23.401026 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:23.401026 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:35:23.401026 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:23.401026 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:35:23.401026 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:35:23.401026 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:35:23.401026 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:35:23.401026 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:23.401026 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:23.401026 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:23.401026 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:35:23.401026 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:23.401026 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:23.401026 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:23.401026 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:35:23.401026 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:23.401523 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:35:23.401523 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:35:23.401523 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:23.401523 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:35:23.401523 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:23.401523 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:35:23.401523 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:35:23.401523 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:23.401523 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:35:23.401523 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:23.401523 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:35:23.404795 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:35:23.404987 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:35:23.405094 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:35:23.405237 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:35:23.405840 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:35:23.406215 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:35:23.406454 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:23.407811 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:35:23.407811 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:23.407811 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:35:23.407811 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:35:23.407811 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:23.407811 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:35:23.407811 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:23.407811 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:35:23.408170 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:23.408170 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:35:23.408170 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:23.408170 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:35:23.408170 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:35:23.408170 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:35:23.408170 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:35:23.408170 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:23.408170 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:23.408170 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:23.408170 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:35:23.408170 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:23.408580 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:23.408580 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:23.408580 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:35:23.408580 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:23.408580 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:35:23.408580 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:35:23.408580 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:23.408580 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:35:23.408580 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:23.408580 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:35:23.408580 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:35:23.408580 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:23.408580 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:35:23.408901 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:23.408901 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:35:23.409223 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:23.409223 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:35:23.409223 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:35:23.409223 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:35:23.409223 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:35:23.409223 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:35:23.409223 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:23.409223 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:35:23.409223 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:35:23.409223 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:23.409223 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:35:23.409223 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:35:23.409223 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:23.409223 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:23.409223 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:35:23.409223 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:23.410502 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:35:23.410502 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:35:23.410502 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:23.410502 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:23.410502 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:23.410502 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:35:23.410502 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:23.410502 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:23.410502 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:23.410502 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:35:23.410502 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:23.410502 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:35:23.410502 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:35:23.410502 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:23.410502 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:35:23.410502 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:23.410502 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:35:23.410502 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:35:23.410502 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:23.411730 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:35:23.411730 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:23.411730 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:35:27.906291 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:35:27.908668 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:35:27.908870 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:35:27.915131 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:35:27.959361 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:35:27.959752 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:27.960830 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:35:27.960830 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:27.960830 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:35:27.960830 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:35:27.960830 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:27.960830 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:35:27.960830 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:27.960830 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:35:27.962059 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.052 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:27.962059 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:35:27.962059 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:27.962059 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:35:27.962059 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:35:27.962059 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:35:27.962059 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:35:27.962059 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:27.962059 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:27.962059 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:27.962059 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:35:27.962059 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:27.962639 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:27.962639 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:27.962639 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:35:27.962639 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:27.962639 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:35:27.962639 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:35:27.962639 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:27.962639 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:35:27.962639 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:27.962639 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:35:27.962639 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:35:27.962639 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:27.962639 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:35:27.963150 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:27.963150 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:35:27.963246 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:27.963246 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:35:27.963246 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:35:27.963246 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:35:27.963246 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:35:27.963246 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:35:27.963246 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:27.963246 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:35:27.963246 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:35:27.963246 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:27.963246 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:35:27.963246 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:35:27.963246 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:27.963246 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:27.963246 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:35:27.963246 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:27.963936 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:35:27.963936 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:35:27.963936 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:27.963936 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:27.963936 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:27.963936 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:35:27.963936 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:27.963936 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:27.963936 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:27.963936 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:35:27.963936 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:27.963936 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:35:27.963936 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:35:27.963936 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:27.963936 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:35:27.963936 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:27.963936 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:35:27.963936 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:35:27.963936 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:27.964704 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:35:27.964704 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:27.964704 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:35:28.371473 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:35:28.372765 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:35:28.372973 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:35:28.374340 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:35:28.374844 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:28.375958 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:35:28.375958 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:28.375958 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:35:28.375958 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:35:28.375958 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:28.375958 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:35:28.375958 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:28.375958 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:35:28.377414 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:28.377414 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:35:28.377414 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:28.377414 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:35:28.377414 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:35:28.377414 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:35:28.377414 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:35:28.377414 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:28.377414 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:28.377414 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:28.377414 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:35:28.377414 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:28.377829 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:28.377829 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:28.377829 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:35:28.377829 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:28.377829 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:35:28.377829 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:35:28.377829 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:28.377829 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:35:28.377829 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:28.377829 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:35:28.377829 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:35:28.377829 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:28.377829 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:35:28.378314 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:28.378314 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:35:28.379108 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:28.379108 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:35:28.379108 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:35:28.379108 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:35:28.379108 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:35:28.379108 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:35:28.379108 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:28.379108 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:35:28.379108 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:35:28.379108 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:28.379108 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:35:28.379108 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:35:28.379108 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:28.379108 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:28.379108 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:35:28.379108 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:28.380088 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:35:28.380088 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:35:28.380088 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:28.380088 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:28.380088 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:28.380088 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:35:28.380088 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:28.380088 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:28.380088 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:28.380088 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:35:28.380088 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:28.380088 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:35:28.380088 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:35:28.380088 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:28.380088 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:35:28.380088 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:28.380088 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:35:28.380088 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:35:28.380088 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:28.381371 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:35:28.381371 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:28.381371 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:35:28.384546 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:35:28.384854 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:35:28.384964 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:35:28.385094 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:35:28.385700 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:35:28.386059 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:28.386876 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:35:28.386876 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:28.386876 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:35:28.386876 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:35:28.386876 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:28.386876 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:35:28.386876 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:28.386876 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:35:28.387704 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:28.387704 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:35:28.387704 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:35:28.387704 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:35:28.387704 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:28.387704 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:35:28.387704 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:28.387704 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:35:28.387704 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:28.387704 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:35:28.387704 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:28.387704 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:35:28.387704 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:28.387704 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:35:28.387704 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:35:28.388108 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:28.388108 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:35:28.388108 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:28.388108 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:35:28.388108 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:35:28.388108 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:28.388108 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:35:28.388108 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:28.388108 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:35:28.388547 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:28.388547 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:35:28.388547 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:28.388547 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:35:28.388547 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:35:28.388547 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:35:28.388547 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:35:28.388547 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:35:28.388547 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:35:28.388547 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:35:28.388547 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:28.388547 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:28.388910 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:28.388910 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:35:28.388910 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:28.388910 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:28.388910 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:28.388910 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:35:28.388910 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:28.388910 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:35:28.388910 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:35:28.388910 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:28.388910 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:35:28.388910 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:28.388910 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:35:28.388910 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:35:28.389264 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:28.389264 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:35:28.389264 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:28.389264 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:35:28.389817 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:28.389817 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:35:28.389817 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:35:28.389817 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:35:28.389817 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:35:28.389817 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:35:28.389817 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:28.389817 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:35:28.389817 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:35:28.389817 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:28.389817 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:35:28.389817 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:35:28.389817 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:28.389817 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:28.389817 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:35:28.389817 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:28.390791 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:35:28.390791 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:35:28.390791 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:35:28.390791 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:35:28.390791 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:35:28.390791 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:28.390791 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:28.390791 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:28.390791 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:35:28.390791 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:28.390791 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:28.390791 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:28.390791 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:35:28.390791 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:28.390791 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:35:28.390791 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:35:28.390791 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:28.390791 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:35:28.391852 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:28.391852 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:35:28.391852 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:35:28.391852 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:28.391852 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:35:28.391852 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:28.391852 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:35:28.395434 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:35:28.395644 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:35:28.395749 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:35:28.395870 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:35:28.396445 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:35:28.396795 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:28.397602 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:35:28.397602 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:28.397602 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:35:28.397602 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:35:28.397602 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:28.397602 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:35:28.397602 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:28.397602 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:35:28.398388 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:28.398388 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:35:28.398388 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:35:28.398388 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:35:28.398388 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:28.398388 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:35:28.398388 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:28.398388 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:35:28.398388 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:28.398388 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:35:28.398388 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:28.398388 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:35:28.398388 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:28.398388 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:35:28.398388 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:35:28.399165 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:28.399165 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:35:28.399165 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:28.399165 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:35:28.399165 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:35:28.399165 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:28.399165 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:35:28.399165 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:28.399165 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:35:28.399630 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:28.399630 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:35:28.399630 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:28.399630 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:35:28.399630 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:35:28.399630 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:35:28.399630 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:35:28.399630 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:35:28.399630 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:35:28.399630 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:28.399630 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:28.399912 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:28.399912 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:35:28.399912 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:28.399912 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:28.399912 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:28.399912 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:35:28.399912 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:28.399912 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:35:28.399912 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:35:28.399912 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:28.399912 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:35:28.399912 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:28.399912 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:35:28.399912 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:35:28.400295 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:28.400295 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:35:28.400295 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:28.400295 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:35:28.400663 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:28.400663 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:35:28.400663 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:35:28.400663 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:35:28.400663 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:35:28.400663 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:35:28.400663 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:28.400663 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:35:28.400663 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:35:28.400663 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:28.400663 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:35:28.400663 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:35:28.400663 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:28.400663 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:28.400663 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:35:28.400663 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:28.400663 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:35:28.401215 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:35:28.401215 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:35:28.401215 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:35:28.401215 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:28.401215 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:28.401215 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:28.401215 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:35:28.401215 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:28.401215 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:28.401215 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:28.401215 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:35:28.401215 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:28.401215 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:35:28.401215 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:35:28.401215 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:28.401215 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:35:28.401215 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:28.401215 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:35:28.401215 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:35:28.401746 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:28.401746 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:35:28.401746 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:28.401746 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:35:28.405198 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:35:28.405417 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:35:28.405531 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:35:28.405662 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:35:28.406135 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:35:28.406506 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:35:28.406692 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:28.407420 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:35:28.407420 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:28.407420 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:35:28.407420 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:35:28.407420 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:28.407420 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:35:28.407420 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:28.407420 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:35:28.408397 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:28.408397 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:35:28.408397 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:28.408397 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:35:28.408397 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:35:28.408397 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:35:28.408397 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:35:28.408397 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:28.408397 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:28.408397 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:28.408397 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:35:28.408397 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:28.408921 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:28.408921 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:28.408921 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:35:28.408921 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:28.408921 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:35:28.408921 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:35:28.408921 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:28.408921 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:35:28.408921 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:28.408921 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:35:28.408921 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:35:28.408921 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:28.408921 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:35:28.409320 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:28.409320 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:35:28.409408 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:28.409408 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:35:28.409408 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:35:28.409408 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:35:28.409408 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:35:28.409408 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:35:28.409408 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:28.409408 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:35:28.409408 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:35:28.409408 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:28.409408 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:35:28.409408 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:35:28.409408 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:28.409408 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:28.409408 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:35:28.409408 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:28.409912 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:35:28.409912 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:35:28.409912 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:28.409912 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:28.409912 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:28.409912 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:35:28.409912 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:28.409912 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:28.409912 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:28.409912 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:35:28.409912 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:28.409912 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:35:28.409912 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:35:28.409912 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:28.409912 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:35:28.409912 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:28.409912 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:35:28.409912 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:35:28.409912 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:28.410906 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:35:28.410906 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:28.410906 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:35:32.907788 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:35:32.908567 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:35:32.908567 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:35:32.909203 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:35:32.945564 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:35:32.945753 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:32.946477 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:35:32.946477 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:32.946477 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:35:32.946477 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:35:32.946477 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:32.946477 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:35:32.946477 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:32.946477 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:35:32.947502 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.038 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:32.947502 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:35:32.947502 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:32.947502 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:35:32.947502 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:35:32.947502 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:35:32.947502 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:35:32.947502 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:32.947502 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:32.947502 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:32.947502 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:35:32.947502 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:32.948092 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:32.948092 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:32.948092 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:35:32.948092 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:32.948092 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:35:32.948092 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:35:32.948092 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:32.948092 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:35:32.948092 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:32.948092 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:35:32.948092 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:35:32.948092 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:32.948092 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:35:32.948671 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:32.948671 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:35:32.948671 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:32.948671 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:35:32.948671 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:35:32.948671 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:35:32.948671 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:35:32.948671 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:35:32.948671 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:32.948671 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:35:32.948671 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:35:32.948671 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:32.948671 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:35:32.948671 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:35:32.948671 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:32.949377 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:32.949377 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:35:32.949377 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:32.949377 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:35:32.949377 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:35:32.949377 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:32.949377 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:32.949377 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:32.949377 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:35:32.949377 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:32.949377 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:32.949377 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:32.949377 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:35:32.949377 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:32.949377 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:35:32.949377 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:35:32.949377 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:32.949377 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:35:32.949377 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:32.950245 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:35:32.950245 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:35:32.950245 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:32.950245 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:35:32.950245 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:32.950245 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:35:33.372902 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:35:33.373461 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:35:33.373655 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:35:33.374962 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:35:33.375363 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:33.376372 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:35:33.376372 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:33.376372 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:35:33.376372 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:35:33.376372 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:33.376372 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:35:33.376372 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:33.376372 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:35:33.378098 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:33.378098 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:35:33.378098 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:33.378098 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:35:33.378098 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:35:33.378098 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:35:33.378098 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:35:33.378098 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:33.378098 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:33.378098 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:33.378098 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:35:33.378098 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:33.378749 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:33.378749 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:33.378749 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:35:33.378749 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:33.378749 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:35:33.378749 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:35:33.378749 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:33.378749 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:35:33.378749 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:33.378749 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:35:33.378749 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:35:33.378749 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:33.378749 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:35:33.379448 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:33.379448 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:35:33.379925 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:33.379925 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:35:33.379925 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:35:33.379925 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:35:33.379925 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:35:33.379925 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:35:33.379925 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:33.379925 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:35:33.379925 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:35:33.379925 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:33.379925 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:35:33.379925 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:35:33.379925 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:33.379925 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:33.379925 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:35:33.379925 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:33.380705 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:35:33.380705 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:35:33.380705 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:33.380705 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:33.380705 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:33.380705 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:35:33.380705 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:33.380705 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:33.380705 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:33.380705 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:35:33.380705 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:33.380705 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:35:33.380705 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:35:33.380705 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:33.380705 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:35:33.380705 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:33.380705 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:35:33.380705 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:35:33.380705 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:33.381793 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:35:33.381793 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:33.381793 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:35:33.385301 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:35:33.385693 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:35:33.385807 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:35:33.385929 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:35:33.386503 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:35:33.386847 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:33.387772 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:35:33.387772 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:33.387772 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:35:33.387772 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:35:33.387772 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:33.387772 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:35:33.387772 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:33.387772 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:35:33.388989 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:33.388989 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:35:33.388989 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:35:33.388989 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:35:33.388989 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:33.388989 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:35:33.388989 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:33.388989 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:35:33.388989 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:33.388989 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:35:33.388989 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:33.388989 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:35:33.388989 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:33.388989 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:35:33.388989 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:35:33.389480 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:33.389480 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:35:33.389480 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:33.389480 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:35:33.389480 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:35:33.389480 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:33.389480 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:35:33.389480 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:33.389480 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:35:33.390294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:33.390294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:35:33.390294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:33.390294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:35:33.390294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:35:33.390294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:35:33.390294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:35:33.390294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:35:33.390294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:35:33.390294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:35:33.390294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:33.390294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:33.390692 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:33.390692 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:35:33.390692 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:33.390692 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:33.390692 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:33.390692 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:35:33.390692 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:33.390692 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:35:33.390692 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:35:33.390692 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:33.390692 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:35:33.390692 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:33.390692 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:35:33.390692 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:35:33.391195 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:33.391195 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:35:33.391195 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:33.391195 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:35:33.391996 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:33.391996 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:35:33.391996 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:35:33.391996 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:35:33.391996 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:35:33.391996 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:35:33.391996 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:33.391996 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:35:33.391996 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:35:33.391996 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:33.391996 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:35:33.391996 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:35:33.391996 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:33.391996 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:33.391996 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:35:33.391996 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:33.392860 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:35:33.392860 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:35:33.392860 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:35:33.392860 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:35:33.392860 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:35:33.392860 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:33.392860 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:33.392860 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:33.392860 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:35:33.392860 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:33.392860 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:33.392860 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:33.392860 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:35:33.392860 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:33.392860 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:35:33.392860 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:35:33.392860 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:33.392860 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:35:33.393796 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:33.393796 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:35:33.393796 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:35:33.393796 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:33.393796 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:35:33.393796 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:33.393796 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:35:33.395915 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:35:33.396384 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:35:33.396547 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:35:33.396765 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:35:33.397235 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:35:33.397680 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:33.398447 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:35:33.398447 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:33.398447 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:35:33.398447 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:35:33.398447 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:33.398447 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:35:33.398447 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:33.398447 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:35:33.399223 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:33.399223 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:35:33.399223 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:35:33.399223 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:35:33.399223 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:33.399223 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:35:33.399223 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:33.399223 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:35:33.399223 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:33.399223 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:35:33.399223 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:33.399223 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:35:33.399223 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:33.399223 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:35:33.399223 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:35:33.400067 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:33.400067 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:35:33.400067 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:33.400067 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:35:33.400067 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:35:33.400067 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:33.400067 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:35:33.400067 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:33.400067 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:35:33.400067 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:33.400067 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:35:33.400067 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:33.400067 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:35:33.400067 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:35:33.400475 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:35:33.400475 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:35:33.400475 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:35:33.400475 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:35:33.400475 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:33.400475 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:33.400475 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:33.400475 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:35:33.400475 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:33.400475 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:33.400475 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:33.400475 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:35:33.400475 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:33.400842 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:35:33.400842 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:35:33.400842 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:33.400842 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:35:33.400842 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:33.400842 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:35:33.400842 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:35:33.400842 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:33.400842 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:35:33.400842 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:33.400842 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:35:33.401144 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:33.401144 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:35:33.401144 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:35:33.401144 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:35:33.401144 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:35:33.401144 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:35:33.401144 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:33.401144 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:35:33.401144 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:35:33.401144 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:33.401144 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:35:33.401144 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:35:33.401144 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:33.401144 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:33.401144 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:35:33.401144 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:33.401144 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:35:33.401640 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:35:33.401640 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:35:33.401640 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:35:33.401640 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:33.401640 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:33.401640 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:33.401640 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:35:33.401640 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:33.401640 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:33.401640 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:33.401640 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:35:33.401640 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:33.401640 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:35:33.401640 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:35:33.401640 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:33.401640 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:35:33.401640 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:33.401640 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:35:33.401640 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:35:33.402148 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:33.402148 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:35:33.402148 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:33.402148 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:35:33.405685 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:35:33.406006 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:35:33.406205 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:35:33.406382 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:35:33.406732 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:35:33.406953 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:35:33.407108 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:33.407643 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:35:33.407643 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:33.407643 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:35:33.407643 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:35:33.407643 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:33.407643 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:35:33.407643 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:33.407643 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:35:33.408497 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:33.408497 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:35:33.408497 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:33.408497 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:35:33.408497 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:35:33.408497 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:35:33.408497 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:35:33.408497 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:33.408497 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:33.408497 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:33.408497 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:35:33.408497 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:33.408888 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:33.408888 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:33.408888 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:35:33.408888 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:33.408888 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:35:33.408888 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:35:33.408888 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:33.408888 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:35:33.408888 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:33.408888 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:35:33.408888 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:35:33.408888 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:33.408888 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:35:33.409292 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:33.409292 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:35:33.409441 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:33.409441 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:35:33.409441 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:35:33.409441 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:35:33.409441 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:35:33.409441 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:35:33.409441 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:33.409441 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:35:33.409441 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:35:33.409441 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:33.409441 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:35:33.409441 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:35:33.409441 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:33.409441 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:33.409441 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:35:33.409441 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:33.409978 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:35:33.409978 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:35:33.409978 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:33.409978 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:33.409978 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:33.409978 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:35:33.409978 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:33.409978 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:33.409978 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:33.409978 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:35:33.409978 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:33.409978 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:35:33.409978 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:35:33.409978 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:33.409978 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:35:33.409978 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:33.409978 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:35:33.409978 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:35:33.409978 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:33.410580 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:35:33.410580 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:33.410580 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:35:37.909429 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:35:37.909429 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:35:37.909429 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:35:37.910661 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:35:37.943459 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:35:37.943850 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:37.944777 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:35:37.944777 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:37.944777 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:35:37.944777 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:35:37.944777 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:37.944777 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:35:37.944777 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:37.944777 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:35:37.945655 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.035 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:37.945655 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:35:37.945655 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:37.945655 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:35:37.945655 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:35:37.945655 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:35:37.945655 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:35:37.945655 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:37.945655 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:37.945655 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:37.945655 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:35:37.945655 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:37.946368 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:37.946368 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:37.946368 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:35:37.946368 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:37.946368 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:35:37.946368 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:35:37.946368 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:37.946368 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:35:37.946368 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:37.946368 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:35:37.946368 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:35:37.946368 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:37.946368 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:35:37.947015 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:37.947015 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:35:37.947015 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:37.947015 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:35:37.947015 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:35:37.947015 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:35:37.947015 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:35:37.947015 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:35:37.947015 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:37.947015 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:35:37.947015 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:35:37.947015 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:37.947015 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:35:37.947015 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:35:37.947015 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:37.947748 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:37.947748 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:35:37.947748 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:37.947748 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:35:37.947748 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:35:37.947748 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:37.947748 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:37.947748 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:37.947748 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:35:37.947748 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:37.947748 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:37.947748 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:37.947748 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:35:37.947748 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:37.947748 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:35:37.947748 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:35:37.947748 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:37.947748 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:35:37.947748 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:37.948629 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:35:37.948629 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:35:37.948629 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:37.948629 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:35:37.948629 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:37.948629 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:35:38.373738 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:35:38.374527 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:35:38.374814 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:35:38.376184 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:35:38.376510 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:38.377270 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:35:38.377270 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:38.377270 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:35:38.377270 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:35:38.377270 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:38.377270 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:35:38.377270 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:38.377270 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:35:38.379149 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:38.379149 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:35:38.379149 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:38.379149 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:35:38.379149 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:35:38.379149 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:35:38.379149 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:35:38.379149 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:38.379149 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:38.379149 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:38.379149 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:35:38.379149 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:38.379830 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:38.379830 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:38.379830 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:35:38.379830 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:38.379830 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:35:38.379830 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:35:38.379830 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:38.379830 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:35:38.379830 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:38.379830 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:35:38.379830 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:35:38.379830 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:38.379830 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:35:38.380454 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:38.380454 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:35:38.381213 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:38.381213 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:35:38.381213 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:35:38.381213 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:35:38.381213 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:35:38.381213 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:35:38.381213 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:38.381213 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:35:38.381213 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:35:38.381213 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:38.381213 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:35:38.381213 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:35:38.381213 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:38.381213 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:38.381213 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:35:38.381213 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:38.382017 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:35:38.382017 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:35:38.382017 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:38.382017 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:38.382017 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:38.382017 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:35:38.382017 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:38.382017 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:38.382017 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:38.382017 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:35:38.382017 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:38.382017 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:35:38.382017 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:35:38.382017 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:38.382017 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:35:38.382017 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:38.382017 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:35:38.382017 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:35:38.382017 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:38.383003 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:35:38.383003 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:38.383003 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:35:38.386024 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:35:38.386497 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:35:38.386599 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:35:38.386779 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:35:38.387284 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:35:38.387550 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:38.388344 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:35:38.388344 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:38.388344 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:35:38.388344 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:35:38.388344 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:38.388344 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:35:38.388344 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:38.388344 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:35:38.389355 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:38.389355 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:35:38.389355 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:35:38.389355 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:35:38.389355 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:38.389355 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:35:38.389355 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:38.389355 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:35:38.389355 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:38.389355 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:35:38.389355 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:38.389355 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:35:38.389355 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:38.389355 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:35:38.389355 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:35:38.389821 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:38.389821 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:35:38.389821 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:38.389821 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:35:38.389821 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:35:38.389821 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:38.389821 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:35:38.389821 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:38.389821 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:35:38.390559 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:38.390559 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:35:38.390559 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:38.390559 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:35:38.390559 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:35:38.390559 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:35:38.390559 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:35:38.390559 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:35:38.390559 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:35:38.390559 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:35:38.390559 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:38.390559 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:38.390982 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:38.390982 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:35:38.390982 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:38.390982 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:38.390982 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:38.390982 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:35:38.390982 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:38.390982 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:35:38.390982 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:35:38.390982 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:38.390982 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:35:38.390982 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:38.390982 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:35:38.390982 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:35:38.391394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:38.391394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:35:38.391394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:38.391394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:35:38.392010 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:38.392010 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:35:38.392010 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:35:38.392010 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:35:38.392010 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:35:38.392010 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:35:38.392010 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:38.392010 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:35:38.392010 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:35:38.392010 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:38.392010 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:35:38.392010 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:35:38.392010 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:38.392010 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:38.392010 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:35:38.392010 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:38.392566 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:35:38.392566 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:35:38.392566 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:35:38.392566 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:35:38.392566 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:35:38.392566 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:38.392566 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:38.392566 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:38.392566 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:35:38.392566 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:38.392566 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:38.392566 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:38.392566 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:35:38.392566 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:38.392566 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:35:38.392566 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:35:38.392566 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:38.392566 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:35:38.393092 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:38.393092 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:35:38.393092 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:35:38.393092 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:38.393092 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:35:38.393092 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:38.393092 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:35:38.396506 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:35:38.396863 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:35:38.397032 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:35:38.397226 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:35:38.397844 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:35:38.398296 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:38.399253 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:35:38.399253 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:38.399253 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:35:38.399253 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:35:38.399253 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:38.399253 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:35:38.399253 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:38.399253 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:35:38.400298 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:38.400298 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:35:38.400298 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:35:38.400298 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:35:38.400298 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:38.400298 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:35:38.400298 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:38.400298 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:35:38.400298 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:38.400298 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:35:38.400298 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:38.400298 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:35:38.400298 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:38.400298 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:35:38.400298 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:35:38.400932 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:38.400932 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:35:38.400932 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:38.400932 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:35:38.400932 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:35:38.400932 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:38.400932 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:35:38.400932 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:38.400932 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:35:38.401629 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:38.401629 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:35:38.401629 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:38.401629 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:35:38.401629 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:35:38.401629 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:35:38.401629 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:35:38.401629 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:35:38.401629 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:35:38.401629 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:38.401629 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:38.402118 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:38.402118 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:35:38.402118 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:38.402118 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:38.402118 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:38.402118 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:35:38.402118 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:38.402118 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:35:38.402118 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:35:38.402118 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:38.402118 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:35:38.402118 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:38.402118 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:35:38.402118 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:35:38.402738 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:38.402738 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:35:38.402738 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:38.402738 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:35:38.403220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:38.403220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:35:38.403220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:35:38.403220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:35:38.403220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:35:38.403220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:35:38.403220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:38.403220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:35:38.403220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:35:38.403220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:38.403220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:35:38.403220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:35:38.403220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:38.403220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:38.403220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:35:38.403220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:38.403220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:35:38.403767 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:35:38.403767 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:35:38.403767 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:35:38.403767 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:38.403767 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:38.403767 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:38.403767 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:35:38.403767 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:38.403767 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:38.403767 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:38.403767 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:35:38.403767 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:38.403767 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:35:38.403767 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:35:38.403767 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:38.403767 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:35:38.403767 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:38.403767 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:35:38.403767 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:35:38.404360 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:38.404360 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:35:38.404360 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:38.404360 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:35:38.406269 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:35:38.406611 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:35:38.406775 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:35:38.406992 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:35:38.407407 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:35:38.407665 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:35:38.407799 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:38.408627 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:35:38.408627 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:38.408627 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:35:38.408627 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:35:38.408627 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:38.408627 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:35:38.408627 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:38.408627 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:35:38.409716 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:38.409716 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:35:38.409716 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:38.409716 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:35:38.409716 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:35:38.409716 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:35:38.409716 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:35:38.409716 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:38.409716 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:38.409716 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:38.409716 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:35:38.409716 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:38.410251 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:38.410251 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:38.410251 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:35:38.410251 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:38.410251 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:35:38.410251 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:35:38.410251 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:38.410251 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:35:38.410251 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:38.410251 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:35:38.410251 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:35:38.410251 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:38.410251 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:35:38.410870 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:38.410870 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:35:38.411191 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:38.411191 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:35:38.411191 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:35:38.411191 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:35:38.411191 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:35:38.411191 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:35:38.411191 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:38.411191 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:35:38.411191 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:35:38.411191 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:38.411191 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:35:38.411191 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:35:38.411191 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:38.411191 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:38.411191 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:35:38.411191 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:38.411906 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:35:38.411906 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:35:38.411906 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:38.411906 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:38.411906 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:38.411906 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:35:38.411906 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:38.411906 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:38.411906 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:38.411906 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:35:38.411906 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:38.411906 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:35:38.411906 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:35:38.411906 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:38.411906 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:35:38.411906 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:38.411906 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:35:38.411906 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:35:38.411906 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:38.412592 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:35:38.412592 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:38.412592 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:35:43.089382 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:35:43.089382 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:35:43.089382 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:35:43.089382 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:35:43.089382 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:35:43.089382 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:43.090065 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:35:43.090065 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:43.090065 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:35:43.090065 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:35:43.090065 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:43.090065 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:35:43.090065 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:43.090065 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:35:43.090065 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.038 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:43.090065 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:35:43.090065 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:43.090065 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:35:43.090065 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:35:43.090065 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:35:43.090065 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:35:43.090627 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:43.090627 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:43.090627 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:43.090627 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:35:43.090627 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:43.090627 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:43.090627 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:43.090627 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:35:43.090627 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:43.090627 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:35:43.090627 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:35:43.090627 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:43.090627 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:35:43.090627 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:43.091019 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:35:43.091019 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:35:43.091019 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:43.091019 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:35:43.091019 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:43.091019 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:35:43.091019 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:43.091019 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:35:43.091019 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:35:43.091019 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:35:43.091019 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:35:43.091019 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:35:43.091019 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:43.091019 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:35:43.091454 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:35:43.091454 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:43.091454 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:35:43.091454 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:35:43.091454 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:43.091454 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:43.091454 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:35:43.091454 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:43.091454 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:35:43.091454 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:35:43.091454 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:43.091454 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:43.091454 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:43.091454 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:35:43.091454 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:43.091454 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:43.091454 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:43.091454 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:35:43.091454 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:43.091454 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:35:43.091454 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:35:43.091974 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:43.091974 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:35:43.091974 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:43.091974 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:35:43.091974 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:35:43.091974 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:43.091974 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:35:43.091974 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:43.091974 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:35:43.375470 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:35:43.375470 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:35:43.375470 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:35:43.376267 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:35:43.377767 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:43.377767 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:35:43.377767 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:43.377767 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:35:43.377767 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:35:43.377767 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:43.377767 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:35:43.377767 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:43.377767 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:35:43.378492 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:43.378492 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:35:43.378492 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:43.378492 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:35:43.378492 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:35:43.378492 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:35:43.378492 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:35:43.378492 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:43.378492 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:43.378492 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:43.378492 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:35:43.378492 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:43.379015 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:43.379015 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:43.379015 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:35:43.379015 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:43.379015 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:35:43.379015 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:35:43.379015 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:43.379015 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:35:43.379015 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:43.379015 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:35:43.379015 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:35:43.379015 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:43.379015 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:35:43.379580 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:43.379580 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:35:43.379994 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:43.379994 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:35:43.379994 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:35:43.379994 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:35:43.379994 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:35:43.379994 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:35:43.379994 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:43.379994 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:35:43.379994 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:35:43.379994 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:43.379994 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:35:43.379994 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:35:43.379994 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:43.379994 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:43.379994 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:35:43.379994 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:43.380427 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:35:43.380427 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:35:43.380427 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:43.380427 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:43.380427 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:43.380427 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:35:43.380427 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:43.380427 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:43.380427 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:43.380427 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:35:43.380427 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:43.380427 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:35:43.380427 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:35:43.380427 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:43.380427 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:35:43.380427 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:43.380427 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:35:43.380427 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:35:43.380427 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:43.380760 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:35:43.380760 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:43.380760 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:35:43.388385 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:35:43.391360 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:35:43.391360 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:35:43.391360 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:35:43.391360 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:35:43.391360 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:43.391360 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:35:43.391360 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:43.391360 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:35:43.391360 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:35:43.391921 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:43.391921 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:35:43.391921 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:43.391921 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:35:43.392160 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:43.392160 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:35:43.392160 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:35:43.392160 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:35:43.392160 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:43.392160 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:35:43.392160 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:43.392160 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:35:43.392160 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:43.392160 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:35:43.392160 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:43.392160 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:35:43.392160 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:43.392160 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:35:43.392160 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:35:43.392902 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:43.392902 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:35:43.392902 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:43.392902 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:35:43.392902 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:35:43.392902 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:43.392902 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:35:43.392902 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:43.392902 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:35:43.393348 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:43.393348 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:35:43.393348 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:43.393348 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:35:43.393348 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:35:43.393348 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:35:43.393348 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:35:43.393348 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:35:43.393348 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:35:43.393348 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:35:43.393348 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:43.393348 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:43.393909 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:43.393909 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:35:43.393909 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:43.393909 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:43.393909 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:43.393909 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:35:43.393909 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:43.393909 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:35:43.393909 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:35:43.393909 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:43.393909 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:35:43.393909 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:43.393909 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:35:43.393909 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:35:43.394566 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:43.394566 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:35:43.394566 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:43.394566 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:35:43.394566 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:43.394566 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:35:43.394566 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:35:43.394566 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:35:43.394566 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:35:43.394566 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:35:43.394566 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:43.394566 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:35:43.394566 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:35:43.394566 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:43.395204 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:35:43.395204 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:35:43.395204 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:43.395204 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:43.395204 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:35:43.395204 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:43.395204 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:35:43.395204 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:35:43.395204 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:35:43.395204 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:35:43.395204 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:35:43.395204 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:43.395204 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:43.395204 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:43.395204 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:35:43.395204 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:43.395204 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:43.395204 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:43.395204 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:35:43.396134 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:43.396134 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:35:43.396134 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:35:43.396134 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:43.396134 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:35:43.396134 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:43.396134 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:35:43.396134 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:35:43.396134 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:43.396134 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:35:43.396134 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:43.396134 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:35:43.396995 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:35:43.397339 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:35:43.397501 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:35:43.397652 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:35:43.398413 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:35:43.398797 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:43.400608 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:35:43.400608 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:43.400608 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:35:43.400608 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:35:43.400608 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:43.400608 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:35:43.400608 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:43.400608 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:35:43.401495 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:43.401495 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:35:43.401495 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:35:43.401495 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:35:43.401495 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:43.401495 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:35:43.401495 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:43.401495 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:35:43.401495 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:43.401495 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:35:43.401495 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:43.401495 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:35:43.401495 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:43.401495 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:35:43.401495 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:35:43.401771 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:43.401771 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:35:43.401771 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:43.401771 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:35:43.401771 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:35:43.401771 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:43.401771 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:35:43.401771 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:43.401771 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:35:43.402451 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:43.402451 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:35:43.402451 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:43.402451 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:35:43.402451 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:35:43.402451 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:35:43.402451 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:35:43.402451 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:35:43.402451 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:35:43.402451 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:43.402451 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:43.402664 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:43.402664 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:35:43.402664 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:43.402664 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:43.402664 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:43.402664 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:35:43.402664 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:43.402664 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:35:43.402664 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:35:43.402664 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:43.402664 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:35:43.402664 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:43.402664 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:35:43.402664 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:35:43.402920 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:43.402920 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:35:43.402920 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:43.402920 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:35:43.404263 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:43.404263 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:35:43.404263 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:35:43.404263 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:35:43.404263 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:35:43.404263 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:35:43.404263 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:43.404263 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:35:43.404263 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:35:43.404263 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:43.404263 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:35:43.404263 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:35:43.404263 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:43.404263 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:43.404263 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:35:43.404263 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:43.404263 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:35:43.404770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:35:43.404770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:35:43.404770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:35:43.404770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:43.404770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:43.404770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:43.404770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:35:43.404770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:43.404770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:43.404770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:43.404770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:35:43.404770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:43.404770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:35:43.404770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:35:43.404770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:43.404770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:35:43.404770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:43.404770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:35:43.404770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:35:43.405380 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:43.405380 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:35:43.405380 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:43.405380 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:35:43.407204 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:35:43.407204 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:35:43.407204 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:35:43.407602 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:35:43.407784 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:35:43.408016 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:35:43.408111 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:43.408666 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:35:43.408666 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:43.408666 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:35:43.408666 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:35:43.408666 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:43.408666 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:35:43.408666 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:43.408666 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:35:43.409669 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:43.409669 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:35:43.409669 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:43.409669 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:35:43.409669 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:35:43.409669 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:35:43.409669 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:35:43.409669 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:43.409669 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:43.409669 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:43.409669 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:35:43.409669 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:43.410021 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:43.410021 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:43.410021 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:35:43.410021 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:43.410021 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:35:43.410021 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:35:43.410021 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:43.410021 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:35:43.410021 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:43.410021 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:35:43.410021 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:35:43.410021 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:43.410021 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:35:43.410392 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:43.410392 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:35:43.410942 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:43.410942 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:35:43.410942 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:35:43.410942 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:35:43.410942 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:35:43.410942 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:35:43.410942 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:43.410942 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:35:43.410942 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:35:43.410942 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:43.410942 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:35:43.410942 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:35:43.410942 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:43.410942 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:43.410942 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:35:43.410942 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:43.411410 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:35:43.411410 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:35:43.411410 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:43.411410 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:43.411410 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:43.411410 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:35:43.411410 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:43.411410 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:43.411410 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:43.411410 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:35:43.411410 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:43.411410 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:35:43.411410 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:35:43.411410 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:43.411410 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:35:43.411410 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:43.411410 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:35:43.411410 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:35:43.411410 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:43.411975 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:35:43.411975 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:43.411975 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:35:47.910256 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:35:47.911072 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:35:47.911072 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:35:47.911902 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:35:47.913643 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG dbcounter [-] [106123] Writing DB stats neutron:SELECT=260 {{(pid=106123) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} May 18 18:35:47.948117 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:35:47.948525 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:47.949588 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:35:47.949588 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:47.949588 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:35:47.949588 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:35:47.949588 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:47.949588 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:35:47.949588 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:47.949588 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:35:47.950845 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.039 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:47.950845 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:35:47.950845 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:47.950845 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:35:47.950845 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:35:47.950845 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:35:47.950845 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:35:47.950845 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:47.950845 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:47.950845 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:47.950845 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:35:47.950845 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:47.951165 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:47.951165 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:47.951165 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:35:47.951165 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:47.951165 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:35:47.951165 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:35:47.951165 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:47.951165 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:35:47.951165 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:47.951165 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:35:47.951165 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:35:47.951165 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:47.951165 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:35:47.951524 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:47.951524 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:35:47.952067 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:47.952067 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:35:47.952067 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:35:47.952067 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:35:47.952067 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:35:47.952067 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:35:47.952067 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:47.952067 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:35:47.952067 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:35:47.952067 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:47.952067 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:35:47.952067 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:35:47.952067 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:47.952067 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:47.952067 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:35:47.952067 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:47.952491 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:35:47.952491 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:35:47.952491 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:47.952491 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:47.952491 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:47.952491 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:35:47.952491 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:47.952491 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:47.952491 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:47.952491 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:35:47.952491 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:47.952491 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:35:47.952491 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:35:47.952491 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:47.952491 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:35:47.952491 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:47.952491 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:35:47.952491 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:35:47.952491 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:47.952961 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:35:47.952961 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:47.952961 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:35:48.375851 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:35:48.376784 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:35:48.376967 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:35:48.378167 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:35:48.378580 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:48.379586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:35:48.379586 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:48.379586 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:35:48.379586 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:35:48.379586 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:48.379586 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:35:48.379586 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:48.379586 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:35:48.380714 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:48.380714 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:35:48.380714 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:48.380714 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:35:48.380714 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:35:48.380714 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:35:48.380714 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:35:48.380714 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:48.380714 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:48.380714 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:48.380714 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:35:48.380714 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:48.381382 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:48.381382 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:48.381382 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:35:48.381382 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:48.381382 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:35:48.381382 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:35:48.381382 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:48.381382 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:35:48.381382 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:48.381382 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:35:48.381382 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:35:48.381382 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:48.381382 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:35:48.381948 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:48.381948 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:35:48.382275 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:48.382275 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:35:48.382275 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:35:48.382275 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:35:48.382275 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:35:48.382275 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:35:48.382275 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:48.382275 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:35:48.382275 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:35:48.382275 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:48.382275 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:35:48.382275 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:35:48.382275 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:48.382275 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:48.382275 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:35:48.382275 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:48.383022 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:35:48.383022 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:35:48.383022 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:48.383022 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:48.383022 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:48.383022 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:35:48.383022 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:48.383022 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:48.383022 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:48.383022 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:35:48.383022 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:48.383022 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:35:48.383022 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:35:48.383022 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:48.383022 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:35:48.383022 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:48.383022 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:35:48.383022 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:35:48.383022 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:48.383826 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:35:48.383826 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:48.383826 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:35:48.388823 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:35:48.389128 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:35:48.389246 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:35:48.389407 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:35:48.389938 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:35:48.390259 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:48.391105 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:35:48.391105 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:48.391105 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:35:48.391105 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:35:48.391105 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:48.391105 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:35:48.391105 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:48.391105 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:35:48.392000 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:48.392000 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:35:48.392000 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:35:48.392000 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:35:48.392000 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:48.392000 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:35:48.392000 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:48.392000 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:35:48.392000 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:48.392000 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:35:48.392000 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:48.392000 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:35:48.392000 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:48.392000 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:35:48.392000 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:35:48.392563 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:48.392563 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:35:48.392563 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:48.392563 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:35:48.392563 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:35:48.392563 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:48.392563 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:35:48.392563 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:48.392563 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:35:48.392857 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:48.392857 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:35:48.392857 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:48.392857 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:35:48.392857 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:35:48.392857 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:35:48.392857 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:35:48.392857 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:35:48.392857 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:35:48.392857 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:35:48.392857 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:48.392857 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:48.393204 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:48.393204 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:35:48.393204 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:48.393204 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:48.393204 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:48.393204 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:35:48.393204 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:48.393204 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:35:48.393204 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:35:48.393204 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:48.393204 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:35:48.393204 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:48.393204 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:35:48.393204 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:35:48.393595 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:48.393595 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:35:48.393595 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:48.393595 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:35:48.393996 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:48.393996 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:35:48.393996 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:35:48.393996 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:35:48.393996 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:35:48.393996 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:35:48.393996 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:48.393996 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:35:48.393996 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:35:48.393996 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:48.393996 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:35:48.393996 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:35:48.393996 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:48.393996 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:48.393996 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:35:48.393996 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:48.394650 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:35:48.394650 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:35:48.394650 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:35:48.394650 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:35:48.394650 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:35:48.394650 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:48.394650 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:48.394650 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:48.394650 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:35:48.394650 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:48.394650 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:48.394650 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:48.394650 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:35:48.394650 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:48.394650 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:35:48.394650 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:35:48.394650 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:48.394650 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:35:48.395185 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:48.395185 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:35:48.395185 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:35:48.395185 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:48.395185 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:35:48.395185 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:48.395185 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:35:48.397517 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:35:48.397867 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:35:48.397978 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:35:48.398111 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:35:48.398628 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:35:48.399039 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:48.399770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:35:48.399770 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:48.399770 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:35:48.399770 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:35:48.399770 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:48.399770 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:35:48.399770 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:48.399770 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:35:48.400655 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:48.400655 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:35:48.400655 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:35:48.400655 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:35:48.400655 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:48.400655 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:35:48.400655 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:48.400655 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:35:48.400655 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:48.400655 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:35:48.400655 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:48.400655 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:35:48.400655 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:48.400655 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:35:48.400655 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:35:48.401055 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:48.401055 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:35:48.401055 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:48.401055 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:35:48.401055 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:35:48.401055 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:48.401055 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:35:48.401055 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:48.401055 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:35:48.401592 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:48.401592 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:35:48.401592 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:48.401592 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:35:48.401592 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:35:48.401592 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:35:48.401592 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:35:48.401592 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:35:48.401592 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:35:48.401592 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:48.401592 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:48.402584 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:48.402584 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:35:48.402584 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:48.402584 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:48.402584 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:48.402584 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:35:48.402584 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:48.402584 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:35:48.402584 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:35:48.402584 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:48.402584 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:35:48.402584 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:48.402584 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:35:48.402584 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:35:48.403031 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:48.403031 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:35:48.403031 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:48.403031 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:35:48.403257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:48.403257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:35:48.403257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:35:48.403257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:35:48.403257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:35:48.403257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:35:48.403257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:48.403257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:35:48.403257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:35:48.403257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:48.403257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:35:48.403257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:35:48.403257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:48.403257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:48.403257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:35:48.403257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:48.403257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:35:48.403777 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:35:48.403777 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:35:48.403777 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:35:48.403777 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:48.403777 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:48.403777 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:48.403777 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:35:48.403777 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:48.403777 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:48.403777 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:48.403777 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:35:48.403777 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:48.403777 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:35:48.403777 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:35:48.403777 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:48.403777 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:35:48.403777 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:48.403777 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:35:48.403777 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:35:48.404427 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:48.404427 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:35:48.404427 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:48.404427 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:35:48.407360 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:35:48.407679 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:35:48.407766 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:35:48.407910 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:35:48.408451 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:35:48.408807 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:35:48.408999 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:48.409777 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:35:48.409777 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:48.409777 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:35:48.409777 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:35:48.409777 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:48.409777 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:35:48.409777 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:48.409777 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:35:48.410442 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:48.410442 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:35:48.410442 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:48.410442 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:35:48.410442 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:35:48.410442 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:35:48.410442 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:35:48.410442 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:48.410442 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:48.410442 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:48.410442 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:35:48.410442 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:48.410791 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:48.410791 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:48.410791 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:35:48.410791 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:48.410791 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:35:48.410791 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:35:48.410791 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:48.410791 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:35:48.410791 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:48.410791 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:35:48.410791 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:35:48.410791 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:48.410791 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:35:48.411152 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:48.411152 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:35:48.411152 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:48.411152 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:35:48.411152 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:35:48.411152 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:35:48.411152 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:35:48.411152 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:35:48.411152 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:48.411152 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:35:48.411152 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:35:48.411152 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:48.411152 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:35:48.411152 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:35:48.411152 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:48.411571 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:48.411571 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:35:48.411571 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:48.411571 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:35:48.411571 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:35:48.411571 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:48.411571 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:48.411571 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:48.411571 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:35:48.411571 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:48.411571 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:48.411571 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:48.411571 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:35:48.411571 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:48.411571 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:35:48.411571 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:35:48.411571 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:48.411571 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:35:48.411571 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:48.412486 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:35:48.412486 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:35:48.412486 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:48.412486 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:35:48.412486 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:48.412486 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:35:52.911173 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:35:52.911924 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:35:52.912116 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:35:52.913524 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:35:52.948800 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:35:52.949203 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:52.950235 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:35:52.950235 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:52.950235 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:35:52.950235 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:35:52.950235 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:52.950235 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:35:52.950235 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:52.950235 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:35:52.951147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.038 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:52.951147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:35:52.951147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:52.951147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:35:52.951147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:35:52.951147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:35:52.951147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:35:52.951147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:52.951147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:52.951147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:52.951147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:35:52.951147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:52.951772 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:52.951772 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:52.951772 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:35:52.951772 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:52.951772 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:35:52.951772 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:35:52.951772 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:52.951772 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:35:52.951772 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:52.951772 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:35:52.951772 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:35:52.951772 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:52.951772 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:35:52.952315 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:52.952315 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:35:52.952315 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:52.952315 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:35:52.952315 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:35:52.952315 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:35:52.952315 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:35:52.952315 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:35:52.952315 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:52.952315 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:35:52.952315 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:35:52.952315 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:52.952315 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:35:52.952315 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:35:52.952315 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:52.952946 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:52.952946 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:35:52.952946 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:52.952946 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:35:52.952946 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:35:52.952946 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:52.952946 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:52.952946 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:52.952946 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:35:52.952946 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:52.952946 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:52.952946 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:52.952946 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:35:52.952946 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:52.952946 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:35:52.952946 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:35:52.952946 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:52.952946 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:35:52.952946 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:52.953734 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:35:52.953734 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:35:52.953734 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:52.953734 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:35:52.953734 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:52.953734 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:35:53.376777 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:35:53.377631 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:35:53.377830 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:35:53.379424 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:35:53.379811 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:53.380823 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:35:53.380823 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:53.380823 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:35:53.380823 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:35:53.380823 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:53.380823 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:35:53.380823 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:53.380823 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:35:53.382548 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:53.382548 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:35:53.382548 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:53.382548 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:35:53.382548 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:35:53.382548 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:35:53.382548 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:35:53.382548 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:53.382548 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:53.382548 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:53.382548 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:35:53.382548 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:53.383094 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:53.383094 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:53.383094 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:35:53.383094 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:53.383094 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:35:53.383094 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:35:53.383094 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:53.383094 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:35:53.383094 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:53.383094 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:35:53.383094 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:35:53.383094 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:53.383094 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:35:53.383709 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:53.383709 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:35:53.384227 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:53.384227 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:35:53.384227 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:35:53.384227 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:35:53.384227 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:35:53.384227 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:35:53.384227 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:53.384227 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:35:53.384227 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:35:53.384227 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:53.384227 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:35:53.384227 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:35:53.384227 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:53.384227 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:53.384227 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:35:53.384227 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:53.384931 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:35:53.384931 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:35:53.384931 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:53.384931 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:53.384931 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:53.384931 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:35:53.384931 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:53.384931 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:53.384931 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:53.384931 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:35:53.384931 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:53.384931 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:35:53.384931 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:35:53.384931 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:53.384931 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:35:53.384931 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:53.384931 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:35:53.384931 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:35:53.384931 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:53.385844 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:35:53.385844 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:53.385844 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:35:53.389300 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:35:53.389715 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:35:53.389829 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:35:53.389954 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:35:53.390497 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:35:53.390924 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:53.391657 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:35:53.391657 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:53.391657 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:35:53.391657 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:35:53.391657 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:53.391657 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:35:53.391657 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:53.391657 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:35:53.392467 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:53.392467 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:35:53.392467 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:35:53.392467 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:35:53.392467 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:53.392467 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:35:53.392467 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:53.392467 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:35:53.392467 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:53.392467 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:35:53.392467 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:53.392467 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:35:53.392467 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:53.392467 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:35:53.392467 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:35:53.392859 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:53.392859 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:35:53.392859 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:53.392859 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:35:53.392859 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:35:53.392859 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:53.392859 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:35:53.392859 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:53.392859 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:35:53.393331 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:53.393331 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:35:53.393331 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:53.393331 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:35:53.393331 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:35:53.393331 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:35:53.393331 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:35:53.393331 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:35:53.393331 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:35:53.393331 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:35:53.393331 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:53.393331 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:53.393639 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:53.393639 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:35:53.393639 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:53.393639 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:53.393639 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:53.393639 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:35:53.393639 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:53.393639 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:35:53.393639 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:35:53.393639 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:53.393639 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:35:53.393639 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:53.393639 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:35:53.393639 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:35:53.394025 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:53.394025 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:35:53.394025 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:53.394025 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:35:53.394386 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:53.394386 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:35:53.394386 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:35:53.394386 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:35:53.394386 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:35:53.394386 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:35:53.394386 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:53.394386 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:35:53.394386 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:35:53.394386 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:53.394386 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:35:53.394386 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:35:53.394386 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:53.394386 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:53.394386 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:35:53.394386 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:53.394893 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:35:53.394893 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:35:53.394893 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:35:53.394893 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:35:53.394893 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:35:53.394893 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:53.394893 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:53.394893 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:53.394893 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:35:53.394893 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:53.394893 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:53.394893 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:53.394893 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:35:53.394893 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:53.394893 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:35:53.394893 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:35:53.394893 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:53.394893 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:35:53.395366 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:53.395366 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:35:53.395366 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:35:53.395366 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:53.395366 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:35:53.395366 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:53.395366 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:35:53.398046 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:35:53.398496 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:35:53.398645 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:35:53.398793 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:35:53.399307 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:35:53.399688 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:53.400449 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:35:53.400449 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:53.400449 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:35:53.400449 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:35:53.400449 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:53.400449 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:35:53.400449 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:53.400449 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:35:53.401232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:53.401232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:35:53.401232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:35:53.401232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:35:53.401232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:53.401232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:35:53.401232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:53.401232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:35:53.401232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:53.401232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:35:53.401232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:53.401232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:35:53.401232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:53.401232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:35:53.401232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:35:53.401687 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:53.401687 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:35:53.401687 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:53.401687 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:35:53.401687 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:35:53.401687 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:53.401687 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:35:53.401687 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:53.401687 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:35:53.402226 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:53.402226 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:35:53.402226 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:53.402226 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:35:53.402226 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:35:53.402226 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:35:53.402226 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:35:53.402226 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:35:53.402226 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:35:53.402226 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:53.402226 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:53.402547 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:53.402547 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:35:53.402547 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:53.402547 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:53.402547 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:53.402547 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:35:53.402547 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:53.402547 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:35:53.402547 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:35:53.402547 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:53.402547 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:35:53.402547 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:53.402547 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:35:53.402547 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:35:53.402916 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:53.402916 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:35:53.402916 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:53.402916 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:35:53.403344 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:53.403344 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:35:53.403344 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:35:53.403344 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:35:53.403344 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:35:53.403344 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:35:53.403344 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:53.403344 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:35:53.403344 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:35:53.403344 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:53.403344 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:35:53.403344 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:35:53.403344 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:53.403344 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:53.403344 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:35:53.403344 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:53.403344 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:35:53.403794 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:35:53.403794 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:35:53.403794 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:35:53.403794 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:53.403794 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:53.403794 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:53.403794 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:35:53.403794 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:53.403794 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:53.403794 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:53.403794 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:35:53.403794 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:53.403794 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:35:53.403794 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:35:53.403794 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:53.403794 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:35:53.403794 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:53.403794 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:35:53.403794 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:35:53.404265 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:53.404265 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:35:53.404265 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:53.404265 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:35:53.407862 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:35:53.408228 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:35:53.408349 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:35:53.408535 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:35:53.408991 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:35:53.409390 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:35:53.409570 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:53.410282 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:35:53.410282 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:53.410282 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:35:53.410282 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:35:53.410282 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:53.410282 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:35:53.410282 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:53.410282 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:35:53.411145 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:53.411145 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:35:53.411145 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:53.411145 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:35:53.411145 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:35:53.411145 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:35:53.411145 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:35:53.411145 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:53.411145 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:53.411145 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:53.411145 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:35:53.411145 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:53.411588 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:53.411588 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:53.411588 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:35:53.411588 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:53.411588 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:35:53.411588 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:35:53.411588 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:53.411588 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:35:53.411588 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:53.411588 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:35:53.411588 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:35:53.411588 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:53.411588 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:35:53.412167 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:53.412167 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:35:53.412342 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:53.412342 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:35:53.412342 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:35:53.412342 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:35:53.412342 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:35:53.412342 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:35:53.412342 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:53.412342 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:35:53.412342 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:35:53.412342 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:53.412342 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:35:53.412342 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:35:53.412342 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:53.412342 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:53.412342 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:35:53.412342 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:53.412887 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:35:53.412887 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:35:53.412887 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:53.412887 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:53.412887 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:53.412887 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:35:53.412887 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:53.412887 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:53.412887 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:53.412887 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:35:53.412887 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:53.412887 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:35:53.412887 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:35:53.412887 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:53.412887 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:35:53.412887 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:53.412887 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:35:53.412887 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:35:53.412887 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:53.413588 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:35:53.413588 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:53.413588 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:35:57.912149 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:35:57.913562 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:35:57.913562 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:35:57.914542 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:35:57.952110 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:35:57.952689 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:57.953809 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:35:57.953809 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:57.953809 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:35:57.953809 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:35:57.953809 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:57.953809 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:35:57.953809 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:57.953809 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:35:57.955118 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.041 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:57.955118 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:35:57.955118 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:57.955118 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:35:57.955118 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:35:57.955118 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:35:57.955118 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:35:57.955118 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:57.955118 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:57.955118 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:57.955118 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:35:57.955118 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:57.955924 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:57.955924 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:57.955924 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:35:57.955924 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:57.955924 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:35:57.955924 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:35:57.955924 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:57.955924 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:35:57.955924 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:57.955924 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:35:57.955924 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:35:57.955924 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:57.955924 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:35:57.956496 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:57.956496 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:35:57.956496 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:57.956496 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:35:57.956496 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:35:57.956496 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:35:57.956496 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:35:57.956496 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:35:57.956496 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:57.956496 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:35:57.956496 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:35:57.956496 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:57.956496 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:35:57.956496 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:35:57.956496 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:57.957393 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:57.957393 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:35:57.957393 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:57.957393 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:35:57.957393 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:35:57.957393 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:57.957393 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:57.957393 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:57.957393 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:35:57.957393 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:57.957393 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:57.957393 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:57.957393 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:35:57.957393 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:57.957393 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:35:57.957393 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:35:57.957393 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:57.957393 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:35:57.957393 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:57.958715 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:35:57.958715 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:35:57.958715 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:57.958715 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:35:57.958715 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:57.958715 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:35:58.377699 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:35:58.378686 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:35:58.378952 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:35:58.380480 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:35:58.380733 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:58.384359 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:35:58.384359 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:58.384359 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:35:58.384359 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:35:58.384359 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:58.384359 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:35:58.384359 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:58.384359 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:35:58.384359 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:58.384359 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:35:58.384359 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:58.384359 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:35:58.384359 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:35:58.384359 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:35:58.384359 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:35:58.385141 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:58.385141 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:58.385141 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:58.385141 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:35:58.385141 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:58.385141 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:58.385141 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:58.385141 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:35:58.385141 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:58.385141 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:35:58.385141 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:35:58.385141 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:58.385141 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:35:58.385141 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:58.385743 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:35:58.385743 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:35:58.385743 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:58.385743 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:35:58.385743 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:58.385743 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:35:58.387586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:58.387586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:35:58.387586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:35:58.387586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:35:58.387586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:35:58.387586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:35:58.387586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:58.387586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:35:58.387586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:35:58.387586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:58.387586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:35:58.387586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:35:58.387586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:58.387586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:58.387586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:35:58.387586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:58.388758 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:35:58.388758 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:35:58.388758 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:58.388758 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:58.388758 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:58.388758 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:35:58.388758 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:58.388758 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:58.388758 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:58.388758 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:35:58.388758 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:58.388758 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:35:58.388758 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:35:58.388758 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:58.388758 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:35:58.388758 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:58.388758 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:35:58.388758 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:35:58.388758 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:58.390554 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:35:58.390554 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:58.390554 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:35:58.390554 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:35:58.390554 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:35:58.390554 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:35:58.390554 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:35:58.391290 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:35:58.391290 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:58.394382 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:35:58.394382 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:58.394382 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:35:58.394382 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:35:58.394382 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:58.394382 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:35:58.394382 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:58.394382 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:35:58.394382 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:58.394382 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:35:58.394382 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:35:58.394382 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:35:58.394382 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:58.394382 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:35:58.394382 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:58.394382 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:35:58.394382 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:58.394382 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:35:58.395294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:58.395294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:35:58.395294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:58.395294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:35:58.395294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:35:58.395294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:58.395294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:35:58.395294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:58.395294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:35:58.395294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:35:58.395294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:58.395294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:35:58.395294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:58.395294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:35:58.395294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:58.396056 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:35:58.396056 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:58.396056 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:35:58.396056 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:35:58.396056 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:35:58.396056 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:35:58.396056 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:35:58.396056 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:35:58.396056 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:35:58.396056 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:58.396056 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:58.396056 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:58.396056 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:35:58.396669 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:58.396669 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:58.396669 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:58.396669 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:35:58.396669 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:58.396669 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:35:58.396669 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:35:58.396669 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:58.396669 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:35:58.396669 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:58.396669 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:35:58.396669 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:35:58.396669 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:58.396669 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:35:58.397262 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:58.397262 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:35:58.397262 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:58.397262 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:35:58.397262 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:35:58.397262 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:35:58.397262 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:35:58.397262 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:35:58.397262 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:58.397262 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:35:58.397262 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:35:58.397262 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:58.397262 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:35:58.397262 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:35:58.397262 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:58.397936 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:58.397936 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:35:58.397936 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:58.397936 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:35:58.397936 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:35:58.397936 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:35:58.397936 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:35:58.397936 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:35:58.397936 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:58.397936 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:58.397936 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:58.397936 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:35:58.397936 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:58.397936 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:58.397936 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:58.397936 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:35:58.397936 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:58.397936 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:35:58.397936 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:35:58.398985 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:58.398985 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:35:58.398985 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:58.398985 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:35:58.398985 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:35:58.398985 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:58.398985 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:35:58.398985 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:58.398985 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:35:58.398985 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:35:58.398985 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:35:58.398985 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:35:58.399559 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:35:58.399559 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:35:58.399735 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:58.400278 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:35:58.400278 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:58.400278 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:35:58.400278 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:35:58.400278 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:58.400278 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:35:58.400278 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:58.400278 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:35:58.400925 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:58.400925 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:35:58.400925 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:35:58.400925 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:35:58.400925 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:58.400925 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:35:58.400925 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:58.400925 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:35:58.400925 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:58.400925 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:35:58.400925 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:58.400925 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:35:58.400925 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:58.400925 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:35:58.400925 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:35:58.401523 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:58.401523 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:35:58.401523 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:58.401523 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:35:58.401523 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:35:58.401523 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:58.401523 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:35:58.401523 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:58.401523 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:35:58.401869 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:58.401869 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:35:58.401869 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:58.401869 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:35:58.401869 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:35:58.401869 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:35:58.401869 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:35:58.401869 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:35:58.401869 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:35:58.401869 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:58.401869 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:58.402484 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:58.402484 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:35:58.402484 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:58.402484 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:58.402484 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:58.402484 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:35:58.402484 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:58.402484 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:35:58.402484 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:35:58.402484 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:58.402484 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:35:58.402484 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:58.402484 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:35:58.402484 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:35:58.403072 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:58.403072 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:35:58.403072 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:58.403072 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:35:58.403072 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:58.403072 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:35:58.403072 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:35:58.403072 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:35:58.403072 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:35:58.403072 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:35:58.403072 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:58.403072 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:35:58.403072 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:35:58.403072 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:58.403586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:35:58.403586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:35:58.403586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:58.403586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:58.403586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:35:58.403586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:58.403586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:35:58.403586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:35:58.403586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:35:58.403586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:35:58.403586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:58.403586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:58.403586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:58.403586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:35:58.403586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:58.403586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:58.403586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:58.403586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:35:58.403586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:58.404144 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:35:58.404144 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:35:58.404144 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:58.404144 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:35:58.404144 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:58.404144 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:35:58.404144 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:35:58.404144 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:58.404144 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:35:58.404144 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:58.404144 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:35:58.408331 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:35:58.408618 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:35:58.408751 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:35:58.408939 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:35:58.409310 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:35:58.409562 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:35:58.409689 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:58.410227 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:35:58.410227 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:58.410227 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:35:58.410227 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:35:58.410227 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:58.410227 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:35:58.410227 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:58.410227 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:35:58.410910 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:58.410910 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:35:58.410910 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:58.410910 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:35:58.410910 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:35:58.410910 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:35:58.410910 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:35:58.410910 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:58.410910 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:58.410910 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:58.410910 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:35:58.410910 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:58.411240 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:35:58.411240 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:58.411240 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:35:58.411240 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:58.411240 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:35:58.411240 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:35:58.411240 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:58.411240 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:35:58.411240 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:58.411240 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:35:58.411240 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:35:58.411240 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:58.411240 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:35:58.411631 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:58.411631 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:35:58.411697 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:58.411697 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:35:58.411697 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:35:58.411697 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:35:58.411697 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:35:58.411697 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:35:58.411697 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:58.411697 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:35:58.411697 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:35:58.411697 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:58.411697 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:35:58.411697 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:35:58.411697 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:35:58.411697 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:35:58.411697 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:35:58.411697 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:35:58.412133 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:35:58.412133 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:35:58.412133 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:58.412133 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:58.412133 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:35:58.412133 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:35:58.412133 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:35:58.412133 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:35:58.412133 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:35:58.412133 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:35:58.412133 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:35:58.412133 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:35:58.412133 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:35:58.412133 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:35:58.412133 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:35:58.412133 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:35:58.412133 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:35:58.412133 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:35:58.412133 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:35:58.412674 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:35:58.412674 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:35:58.412674 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:36:02.912883 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:36:02.913961 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:36:02.913961 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:36:02.915206 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:36:02.950464 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:36:02.950932 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:02.952040 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:36:02.952040 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:02.952040 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:36:02.952040 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:36:02.952040 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:02.952040 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:36:02.952040 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:02.952040 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:36:02.953353 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.038 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:02.953353 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:36:02.953353 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:02.953353 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:36:02.953353 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:36:02.953353 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:36:02.953353 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:36:02.953353 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:02.953353 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:02.953353 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:02.953353 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:36:02.953353 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:02.954203 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:02.954203 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:02.954203 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:36:02.954203 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:02.954203 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:36:02.954203 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:36:02.954203 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:02.954203 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:36:02.954203 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:02.954203 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:36:02.954203 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:36:02.954203 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:02.954203 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:36:02.954961 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:02.954961 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:36:02.954961 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:02.954961 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:36:02.954961 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:36:02.954961 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:36:02.954961 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:36:02.954961 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:36:02.954961 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:02.954961 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:36:02.954961 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:36:02.954961 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:02.954961 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:36:02.954961 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:36:02.954961 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:02.955863 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:02.955863 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:36:02.955863 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:02.955863 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:36:02.955863 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:36:02.955863 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:02.955863 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:02.955863 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:02.955863 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:36:02.955863 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:02.955863 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:02.955863 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:02.955863 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:36:02.955863 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:02.955863 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:36:02.955863 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:36:02.955863 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:02.955863 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:36:02.955863 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:02.956949 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:36:02.956949 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:36:02.956949 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:02.956949 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:36:02.956949 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:02.956949 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:36:03.378765 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:36:03.379549 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:36:03.379724 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:36:03.381083 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:36:03.381496 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:03.382742 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:36:03.382742 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:03.382742 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:36:03.382742 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:36:03.382742 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:03.382742 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:36:03.382742 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:03.382742 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:36:03.384544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:03.384544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:36:03.384544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:03.384544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:36:03.384544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:36:03.384544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:36:03.384544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:36:03.384544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:03.384544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:03.384544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:03.384544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:36:03.384544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:03.385406 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:03.385406 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:03.385406 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:36:03.385406 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:03.385406 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:36:03.385406 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:36:03.385406 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:03.385406 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:36:03.385406 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:03.385406 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:36:03.385406 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:36:03.385406 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:03.385406 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:36:03.386017 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:03.386017 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:36:03.386294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:03.386294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:36:03.386294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:36:03.386294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:36:03.386294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:36:03.386294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:36:03.386294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:03.386294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:36:03.386294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:36:03.386294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:03.386294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:36:03.386294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:36:03.386294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:03.386294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:03.386294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:36:03.386294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:03.386758 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:36:03.386758 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:36:03.386758 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:03.386758 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:03.386758 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:03.386758 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:36:03.386758 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:03.386758 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:03.386758 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:03.386758 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:36:03.386758 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:03.386758 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:36:03.386758 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:36:03.386758 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:03.386758 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:36:03.386758 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:03.386758 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:36:03.386758 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:36:03.386758 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:03.387254 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:36:03.387254 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:03.387254 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:36:03.390301 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:36:03.390660 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:36:03.390779 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:36:03.390908 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:36:03.391387 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:36:03.391615 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:03.392171 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:36:03.392171 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:03.392171 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:36:03.392171 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:36:03.392171 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:03.392171 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:36:03.392171 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:03.392171 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:36:03.393042 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:03.393042 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:36:03.393042 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:36:03.393042 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:36:03.393042 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:03.393042 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:36:03.393042 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:03.393042 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:36:03.393042 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:03.393042 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:36:03.393042 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:03.393042 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:36:03.393042 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:03.393042 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:36:03.393042 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:36:03.393635 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:03.393635 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:36:03.393635 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:03.393635 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:36:03.393635 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:36:03.393635 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:03.393635 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:36:03.393635 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:03.393635 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:36:03.393985 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:03.393985 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:36:03.393985 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:03.393985 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:36:03.393985 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:36:03.393985 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:36:03.393985 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:36:03.393985 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:36:03.393985 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:36:03.393985 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:36:03.393985 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:03.393985 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:03.394459 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:03.394459 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:36:03.394459 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:03.394459 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:03.394459 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:03.394459 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:36:03.394459 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:03.394459 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:36:03.394459 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:36:03.394459 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:03.394459 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:36:03.394459 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:03.394459 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:36:03.394459 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:36:03.395106 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:03.395106 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:36:03.395106 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:03.395106 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:36:03.395106 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:03.395106 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:36:03.395106 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:36:03.395106 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:36:03.395106 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:36:03.395106 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:36:03.395106 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:03.395106 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:36:03.395106 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:36:03.395106 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:03.395779 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:36:03.395779 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:36:03.395779 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:03.395779 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:03.395779 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:36:03.395779 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:03.395779 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:36:03.395779 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:36:03.395779 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:36:03.395779 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:36:03.395779 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:36:03.395779 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:03.395779 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:03.395779 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:03.395779 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:36:03.395779 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:03.395779 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:03.395779 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:03.395779 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:36:03.396639 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:03.396639 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:36:03.396639 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:36:03.396639 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:03.396639 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:36:03.396639 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:03.396639 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:36:03.396639 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:36:03.396639 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:03.396639 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:36:03.396639 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:03.396639 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:36:03.399115 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:36:03.399429 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:36:03.399547 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:36:03.399688 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:36:03.400256 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:36:03.400652 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:03.401445 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:36:03.401445 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:03.401445 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:36:03.401445 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:36:03.401445 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:03.401445 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:36:03.401445 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:03.401445 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:36:03.402228 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:03.402228 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:36:03.402228 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:36:03.402228 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:36:03.402228 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:03.402228 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:36:03.402228 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:03.402228 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:36:03.402228 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:03.402228 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:36:03.402228 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:03.402228 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:36:03.402228 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:03.402228 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:36:03.402228 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:36:03.402681 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:03.402681 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:36:03.402681 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:03.402681 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:36:03.402681 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:36:03.402681 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:03.402681 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:36:03.402681 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:03.402681 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:36:03.403111 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:03.403111 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:36:03.403111 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:03.403111 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:36:03.403111 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:36:03.403111 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:36:03.403111 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:36:03.403111 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:36:03.403111 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:36:03.403111 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:03.403111 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:03.403449 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:03.403449 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:36:03.403449 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:03.403449 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:03.403449 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:03.403449 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:36:03.403449 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:03.403449 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:36:03.403449 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:36:03.403449 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:03.403449 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:36:03.403449 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:03.403449 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:36:03.403449 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:36:03.403865 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:03.403865 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:36:03.403865 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:03.403865 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:36:03.404203 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:03.404203 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:36:03.404203 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:36:03.404203 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:36:03.404203 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:36:03.404203 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:36:03.404203 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:03.404203 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:36:03.404203 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:36:03.404203 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:03.404203 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:36:03.404203 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:36:03.404203 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:03.404203 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:03.404203 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:36:03.404203 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:03.404203 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:36:03.404718 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:36:03.404718 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:36:03.404718 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:36:03.404718 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:03.404718 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:03.404718 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:03.404718 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:36:03.404718 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:03.404718 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:03.404718 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:03.404718 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:36:03.404718 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:03.404718 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:36:03.404718 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:36:03.404718 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:03.404718 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:36:03.404718 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:03.404718 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:36:03.404718 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:36:03.405250 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:03.405250 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:36:03.405250 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:03.405250 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:36:03.408862 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:36:03.409313 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:36:03.409557 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:36:03.409860 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:36:03.410550 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:36:03.410895 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:36:03.411084 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:03.412066 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:36:03.412066 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:03.412066 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:36:03.412066 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:36:03.412066 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:03.412066 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:36:03.412066 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:03.412066 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:36:03.413625 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:03.413625 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:36:03.413625 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:03.413625 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:36:03.413625 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:36:03.413625 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:36:03.413625 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:36:03.413625 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:03.413625 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:03.413625 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:03.413625 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:36:03.413625 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:03.414207 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:03.414207 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:03.414207 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:36:03.414207 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:03.414207 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:36:03.414207 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:36:03.414207 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:03.414207 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:36:03.414207 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:03.414207 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:36:03.414207 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:36:03.414207 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:03.414207 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:36:03.414901 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:03.414901 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:36:03.415419 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:03.415419 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:36:03.415419 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:36:03.415419 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:36:03.415419 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:36:03.415419 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:36:03.415419 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:03.415419 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:36:03.415419 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:36:03.415419 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:03.415419 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:36:03.415419 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:36:03.415419 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:03.415419 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:03.415419 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:36:03.415419 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:03.416221 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:36:03.416221 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:36:03.416221 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:03.416221 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:03.416221 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:03.416221 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:36:03.416221 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:03.416221 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:03.416221 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:03.416221 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:36:03.416221 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:03.416221 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:36:03.416221 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:36:03.416221 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:03.416221 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:36:03.416221 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:03.416221 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:36:03.416221 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:36:03.416221 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:03.417118 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:36:03.417118 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:03.417118 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:36:07.913628 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:36:07.915352 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:36:07.915572 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:36:07.916994 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:36:07.951495 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:36:07.952038 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:07.953271 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:36:07.953271 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:07.953271 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:36:07.953271 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:36:07.953271 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:07.953271 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:36:07.953271 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:07.953271 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:36:07.954463 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.038 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:07.954463 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:36:07.954463 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:07.954463 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:36:07.954463 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:36:07.954463 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:36:07.954463 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:36:07.954463 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:07.954463 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:07.954463 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:07.954463 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:36:07.954463 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:07.954789 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:07.954789 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:07.954789 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:36:07.954789 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:07.954789 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:36:07.954789 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:36:07.954789 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:07.954789 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:36:07.954789 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:07.954789 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:36:07.954789 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:36:07.954789 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:07.954789 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:36:07.955126 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:07.955126 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:36:07.955727 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:07.955727 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:36:07.955727 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:36:07.955727 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:36:07.955727 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:36:07.955727 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:36:07.955727 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:07.955727 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:36:07.955727 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:36:07.955727 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:07.955727 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:36:07.955727 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:36:07.955727 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:07.955727 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:07.955727 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:36:07.955727 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:07.956688 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:36:07.956688 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:36:07.956688 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:07.956688 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:07.956688 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:07.956688 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:36:07.956688 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:07.956688 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:07.956688 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:07.956688 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:36:07.956688 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:07.956688 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:36:07.956688 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:36:07.956688 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:07.956688 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:36:07.956688 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:07.956688 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:36:07.956688 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:36:07.956688 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:07.957656 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:36:07.957656 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:07.957656 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:36:08.379678 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:36:08.380429 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:36:08.380532 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:36:08.381454 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:36:08.381863 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:08.382946 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:36:08.382946 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:08.382946 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:36:08.382946 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:36:08.382946 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:08.382946 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:36:08.382946 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:08.382946 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:36:08.384698 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:08.384698 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:36:08.384698 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:08.384698 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:36:08.384698 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:36:08.384698 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:36:08.384698 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:36:08.384698 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:08.384698 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:08.384698 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:08.384698 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:36:08.384698 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:08.385428 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:08.385428 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:08.385428 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:36:08.385428 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:08.385428 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:36:08.385428 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:36:08.385428 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:08.385428 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:36:08.385428 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:08.385428 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:36:08.385428 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:36:08.385428 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:08.385428 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:36:08.385905 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:08.385905 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:36:08.386696 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:08.386696 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:36:08.386696 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:36:08.386696 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:36:08.386696 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:36:08.386696 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:36:08.386696 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:08.386696 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:36:08.386696 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:36:08.386696 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:08.386696 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:36:08.386696 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:36:08.386696 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:08.386696 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:08.386696 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:36:08.386696 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:08.387231 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:36:08.387231 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:36:08.387231 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:08.387231 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:08.387231 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:08.387231 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:36:08.387231 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:08.387231 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:08.387231 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:08.387231 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:36:08.387231 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:08.387231 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:36:08.387231 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:36:08.387231 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:08.387231 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:36:08.387231 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:08.387231 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:36:08.387231 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:36:08.387231 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:08.387948 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:36:08.387948 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:08.387948 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:36:08.390769 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:36:08.391089 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:36:08.391245 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:36:08.391435 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:36:08.391934 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:36:08.392281 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:08.392999 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:36:08.392999 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:08.392999 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:36:08.392999 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:36:08.392999 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:08.392999 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:36:08.392999 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:08.392999 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:36:08.393879 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:08.393879 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:36:08.393879 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:36:08.393879 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:36:08.393879 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:08.393879 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:36:08.393879 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:08.393879 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:36:08.393879 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:08.393879 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:36:08.393879 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:08.393879 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:36:08.393879 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:08.393879 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:36:08.393879 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:36:08.394482 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:08.394482 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:36:08.394482 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:08.394482 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:36:08.394482 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:36:08.394482 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:08.394482 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:36:08.394482 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:08.394482 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:36:08.394825 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:08.394825 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:36:08.394825 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:08.394825 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:36:08.394825 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:36:08.394825 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:36:08.394825 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:36:08.394825 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:36:08.394825 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:36:08.394825 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:36:08.394825 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:08.394825 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:08.395261 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:08.395261 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:36:08.395261 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:08.395261 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:08.395261 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:08.395261 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:36:08.395261 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:08.395261 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:36:08.395261 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:36:08.395261 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:08.395261 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:36:08.395261 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:08.395261 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:36:08.395261 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:36:08.395789 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:08.395789 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:36:08.395789 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:08.395789 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:36:08.395789 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:08.395789 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:36:08.395789 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:36:08.395789 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:36:08.395789 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:36:08.395789 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:36:08.395789 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:08.395789 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:36:08.395789 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:36:08.395789 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:08.396346 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:36:08.396346 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:36:08.396346 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:08.396346 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:08.396346 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:36:08.396346 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:08.396346 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:36:08.396346 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:36:08.396346 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:36:08.396346 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:36:08.396346 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:36:08.396346 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:08.396346 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:08.396346 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:08.396346 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:36:08.396346 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:08.396346 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:08.396346 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:08.396346 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:36:08.397067 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:08.397067 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:36:08.397067 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:36:08.397067 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:08.397067 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:36:08.397067 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:08.397067 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:36:08.397067 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:36:08.397067 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:08.397067 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:36:08.397067 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:08.397067 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:36:08.399466 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:36:08.399763 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:36:08.399911 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:36:08.400057 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:36:08.400382 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:36:08.400634 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:08.401138 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:36:08.401138 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:08.401138 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:36:08.401138 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:36:08.401138 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:08.401138 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:36:08.401138 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:08.401138 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:36:08.402048 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:08.402048 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:36:08.402048 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:36:08.402048 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:36:08.402048 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:08.402048 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:36:08.402048 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:08.402048 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:36:08.402048 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:08.402048 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:36:08.402048 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:08.402048 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:36:08.402048 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:08.402048 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:36:08.402048 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:36:08.402536 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:08.402536 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:36:08.402536 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:08.402536 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:36:08.402536 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:36:08.402536 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:08.402536 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:36:08.402536 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:08.402536 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:36:08.403201 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:08.403201 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:36:08.403201 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:08.403201 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:36:08.403201 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:36:08.403201 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:36:08.403201 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:36:08.403201 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:36:08.403201 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:36:08.403201 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:08.403201 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:08.403491 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:08.403491 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:36:08.403491 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:08.403491 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:08.403491 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:08.403491 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:36:08.403491 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:08.403491 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:36:08.403491 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:36:08.403491 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:08.403491 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:36:08.403491 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:08.403491 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:36:08.403491 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:36:08.403846 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:08.403846 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:36:08.403846 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:08.403846 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:36:08.404557 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:08.404557 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:36:08.404557 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:36:08.404557 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:36:08.404557 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:36:08.404557 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:36:08.404557 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:08.404557 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:36:08.404557 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:36:08.404557 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:08.404557 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:36:08.404557 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:36:08.404557 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:08.404557 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:08.404557 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:36:08.404557 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:08.404557 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:36:08.405029 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:36:08.405029 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:36:08.405029 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:36:08.405029 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:08.405029 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:08.405029 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:08.405029 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:36:08.405029 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:08.405029 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:08.405029 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:08.405029 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:36:08.405029 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:08.405029 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:36:08.405029 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:36:08.405029 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:08.405029 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:36:08.405029 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:08.405029 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:36:08.405029 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:36:08.405537 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:08.405537 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:36:08.405537 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:08.405537 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:36:08.409335 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:36:08.409692 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:36:08.409854 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:36:08.410002 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:36:08.410362 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:36:08.410570 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:36:08.410680 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:08.411149 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:36:08.411149 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:08.411149 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:36:08.411149 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:36:08.411149 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:08.411149 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:36:08.411149 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:08.411149 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:36:08.412118 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:08.412118 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:36:08.412118 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:08.412118 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:36:08.412118 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:36:08.412118 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:36:08.412118 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:36:08.412118 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:08.412118 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:08.412118 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:08.412118 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:36:08.412118 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:08.412872 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:08.412872 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:08.412872 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:36:08.412872 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:08.412872 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:36:08.412872 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:36:08.412872 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:08.412872 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:36:08.412872 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:08.412872 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:36:08.412872 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:36:08.412872 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:08.412872 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:36:08.413565 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:08.413565 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:36:08.413565 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:08.413565 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:36:08.413565 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:36:08.413565 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:36:08.413565 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:36:08.413565 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:36:08.413565 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:08.413565 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:36:08.413565 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:36:08.413565 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:08.413565 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:36:08.413565 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:36:08.413565 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:08.414451 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:08.414451 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:36:08.414451 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:08.414451 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:36:08.414451 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:36:08.414451 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:08.414451 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:08.414451 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:08.414451 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:36:08.414451 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:08.414451 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:08.414451 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:08.414451 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:36:08.414451 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:08.414451 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:36:08.414451 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:36:08.414451 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:08.414451 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:36:08.414451 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:08.415359 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:36:08.415359 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:36:08.415359 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:08.415359 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:36:08.415359 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:08.415359 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:36:12.915516 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:36:12.917244 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:36:12.917464 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:36:12.918795 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:36:12.950972 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:36:12.951449 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:12.952492 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:36:12.952492 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:12.952492 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:36:12.952492 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:36:12.952492 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:12.952492 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:36:12.952492 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:12.952492 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:36:12.953598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.035 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:12.953598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:36:12.953598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:12.953598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:36:12.953598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:36:12.953598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:36:12.953598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:36:12.953598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:12.953598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:12.953598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:12.953598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:36:12.953598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:12.954252 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:12.954252 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:12.954252 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:36:12.954252 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:12.954252 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:36:12.954252 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:36:12.954252 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:12.954252 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:36:12.954252 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:12.954252 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:36:12.954252 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:36:12.954252 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:12.954252 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:36:12.955802 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:12.955802 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:36:12.955802 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:12.955802 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:36:12.955802 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:36:12.955802 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:36:12.955802 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:36:12.955802 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:36:12.955802 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:12.955802 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:36:12.955802 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:36:12.955802 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:12.955802 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:36:12.955802 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:36:12.955802 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:12.956901 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:12.956901 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:36:12.956901 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:12.956901 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:36:12.956901 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:36:12.956901 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:12.956901 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:12.956901 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:12.956901 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:36:12.956901 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:12.956901 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:12.956901 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:12.956901 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:36:12.956901 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:12.956901 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:36:12.956901 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:36:12.956901 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:12.956901 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:36:12.956901 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:12.959089 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:36:12.959089 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:36:12.959089 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:12.959089 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:36:12.959089 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:12.959089 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:36:13.380580 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:36:13.381309 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:36:13.381547 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:36:13.382720 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:36:13.382997 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:13.383764 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:36:13.383764 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:13.383764 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:36:13.383764 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:36:13.383764 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:13.383764 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:36:13.383764 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:13.383764 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:36:13.385311 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:13.385311 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:36:13.385311 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:13.385311 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:36:13.385311 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:36:13.385311 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:36:13.385311 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:36:13.385311 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:13.385311 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:13.385311 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:13.385311 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:36:13.385311 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:13.385886 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:13.385886 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:13.385886 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:36:13.385886 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:13.385886 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:36:13.385886 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:36:13.385886 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:13.385886 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:36:13.385886 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:13.385886 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:36:13.385886 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:36:13.385886 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:13.385886 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:36:13.386419 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:13.386419 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:36:13.386996 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:13.386996 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:36:13.386996 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:36:13.386996 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:36:13.386996 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:36:13.386996 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:36:13.386996 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:13.386996 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:36:13.386996 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:36:13.386996 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:13.386996 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:36:13.386996 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:36:13.386996 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:13.386996 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:13.386996 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:36:13.386996 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:13.387648 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:36:13.387648 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:36:13.387648 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:13.387648 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:13.387648 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:13.387648 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:36:13.387648 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:13.387648 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:13.387648 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:13.387648 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:36:13.387648 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:13.387648 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:36:13.387648 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:36:13.387648 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:13.387648 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:36:13.387648 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:13.387648 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:36:13.387648 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:36:13.387648 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:13.388479 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:36:13.388479 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:13.388479 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:36:13.391259 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:36:13.391592 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:36:13.391693 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:36:13.391812 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:36:13.392176 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:36:13.392406 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:13.392921 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:36:13.392921 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:13.392921 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:36:13.392921 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:36:13.392921 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:13.392921 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:36:13.392921 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:13.392921 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:36:13.393706 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:13.393706 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:36:13.393706 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:36:13.393706 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:36:13.393706 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:13.393706 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:36:13.393706 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:13.393706 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:36:13.393706 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:13.393706 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:36:13.393706 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:13.393706 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:36:13.393706 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:13.393706 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:36:13.393706 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:36:13.394244 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:13.394244 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:36:13.394244 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:13.394244 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:36:13.394244 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:36:13.394244 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:13.394244 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:36:13.394244 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:13.394244 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:36:13.394560 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:13.394560 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:36:13.394560 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:13.394560 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:36:13.394560 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:36:13.394560 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:36:13.394560 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:36:13.394560 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:36:13.394560 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:36:13.394560 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:36:13.394560 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:13.394560 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:13.394919 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:13.394919 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:36:13.394919 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:13.394919 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:13.394919 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:13.394919 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:36:13.394919 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:13.394919 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:36:13.394919 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:36:13.394919 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:13.394919 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:36:13.394919 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:13.394919 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:36:13.394919 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:36:13.395433 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:13.395433 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:36:13.395433 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:13.395433 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:36:13.395582 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:13.395582 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:36:13.395582 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:36:13.395582 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:36:13.395582 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:36:13.395582 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:36:13.395582 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:13.395582 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:36:13.395582 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:36:13.395582 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:13.395582 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:36:13.395582 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:36:13.395582 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:13.395582 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:13.395582 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:36:13.395582 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:13.396043 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:36:13.396043 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:36:13.396043 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:36:13.396043 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:36:13.396043 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:36:13.396043 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:13.396043 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:13.396043 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:13.396043 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:36:13.396043 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:13.396043 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:13.396043 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:13.396043 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:36:13.396043 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:13.396043 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:36:13.396043 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:36:13.396043 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:13.396043 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:36:13.396576 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:13.396576 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:36:13.396576 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:36:13.396576 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:13.396576 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:36:13.396576 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:13.396576 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:36:13.399939 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:36:13.400250 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:36:13.400383 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:36:13.400517 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:36:13.400886 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:36:13.401149 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:13.401734 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:36:13.401734 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:13.401734 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:36:13.401734 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:36:13.401734 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:13.401734 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:36:13.401734 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:13.401734 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:36:13.402479 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:13.402479 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:36:13.402479 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:36:13.402479 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:36:13.402479 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:13.402479 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:36:13.402479 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:13.402479 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:36:13.402479 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:13.402479 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:36:13.402479 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:13.402479 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:36:13.402479 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:13.402479 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:36:13.402479 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:36:13.402913 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:13.402913 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:36:13.402913 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:13.402913 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:36:13.402913 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:36:13.402913 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:13.402913 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:36:13.402913 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:13.402913 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:36:13.403307 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:13.403307 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:36:13.403307 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:13.403307 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:36:13.403307 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:36:13.403307 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:36:13.403307 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:36:13.403307 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:36:13.403307 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:36:13.403307 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:13.403307 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:13.403660 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:13.403660 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:36:13.403660 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:13.403660 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:13.403660 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:13.403660 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:36:13.403660 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:13.403660 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:36:13.403660 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:36:13.403660 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:13.403660 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:36:13.403660 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:13.403660 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:36:13.403660 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:36:13.404135 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:13.404135 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:36:13.404135 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:13.404135 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:36:13.404339 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:13.404339 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:36:13.404339 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:36:13.404339 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:36:13.404339 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:36:13.404339 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:36:13.404339 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:13.404339 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:36:13.404339 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:36:13.404339 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:13.404339 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:36:13.404339 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:36:13.404339 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:13.404339 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:13.404339 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:36:13.404339 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:13.404339 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:36:13.404848 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:36:13.404848 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:36:13.404848 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:36:13.404848 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:13.404848 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:13.404848 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:13.404848 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:36:13.404848 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:13.404848 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:13.404848 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:13.404848 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:36:13.404848 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:13.404848 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:36:13.404848 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:36:13.404848 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:13.404848 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:36:13.404848 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:13.404848 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:36:13.404848 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:36:13.405438 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:13.405438 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:36:13.405438 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:13.405438 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:36:13.409908 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:36:13.410199 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:36:13.410307 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:36:13.410479 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:36:13.410827 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:36:13.411029 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:36:13.411143 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:13.411695 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:36:13.411695 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:13.411695 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:36:13.411695 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:36:13.411695 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:13.411695 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:36:13.411695 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:13.411695 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:36:13.412980 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:13.412980 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:36:13.412980 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:13.412980 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:36:13.412980 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:36:13.412980 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:36:13.412980 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:36:13.412980 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:13.412980 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:13.412980 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:13.412980 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:36:13.412980 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:13.413354 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:13.413354 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:13.413354 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:36:13.413354 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:13.413354 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:36:13.413354 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:36:13.413354 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:13.413354 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:36:13.413354 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:13.413354 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:36:13.413354 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:36:13.413354 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:13.413354 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:36:13.413712 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:13.413712 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:36:13.414520 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:13.414520 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:36:13.414520 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:36:13.414520 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:36:13.414520 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:36:13.414520 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:36:13.414520 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:13.414520 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:36:13.414520 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:36:13.414520 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:13.414520 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:36:13.414520 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:36:13.414520 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:13.414520 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:13.414520 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:36:13.414520 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:13.415263 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:36:13.415263 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:36:13.415263 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:13.415263 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:13.415263 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:13.415263 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:36:13.415263 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:13.415263 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:13.415263 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:13.415263 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:36:13.415263 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:13.415263 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:36:13.415263 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:36:13.415263 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:13.415263 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:36:13.415263 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:13.415263 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:36:13.415263 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:36:13.415263 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:13.416271 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:36:13.416271 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:13.416271 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:36:17.917310 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:36:17.918062 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:36:17.918219 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:36:17.919712 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:36:17.952210 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:36:17.952830 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:17.953651 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:36:17.953651 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:17.953651 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:36:17.953651 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:36:17.953651 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:17.953651 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:36:17.953651 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:17.953651 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:36:17.954344 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.035 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:17.954344 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:36:17.954344 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:17.954344 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:36:17.954344 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:36:17.954344 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:36:17.954344 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:36:17.954344 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:17.954344 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:17.954344 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:17.954344 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:36:17.954344 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:17.954678 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:17.954678 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:17.954678 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:36:17.954678 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:17.954678 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:36:17.954678 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:36:17.954678 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:17.954678 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:36:17.954678 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:17.954678 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:36:17.954678 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:36:17.954678 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:17.954678 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:36:17.955022 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:17.955022 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:36:17.955273 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:17.955273 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:36:17.955273 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:36:17.955273 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:36:17.955273 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:36:17.955273 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:36:17.955273 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:17.955273 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:36:17.955273 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:36:17.955273 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:17.955273 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:36:17.955273 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:36:17.955273 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:17.955273 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:17.955273 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:36:17.955273 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:17.955876 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:36:17.955876 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:36:17.955876 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:17.955876 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:17.955876 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:17.955876 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:36:17.955876 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:17.955876 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:17.955876 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:17.955876 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:36:17.955876 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:17.955876 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:36:17.955876 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:36:17.955876 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:17.955876 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:36:17.955876 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:17.955876 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:36:17.955876 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:36:17.955876 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:17.956400 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:36:17.956400 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:17.956400 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:36:18.381647 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:36:18.382316 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:36:18.382529 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:36:18.384902 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:36:18.384902 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:18.385458 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:36:18.385458 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:18.385458 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:36:18.385458 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:36:18.385458 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:18.385458 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:36:18.385458 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:18.385458 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:36:18.387139 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:18.387139 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:36:18.387139 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:18.387139 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:36:18.387139 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:36:18.387139 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:36:18.387139 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:36:18.387139 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:18.387139 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:18.387139 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:18.387139 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:36:18.387139 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:18.387944 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:18.387944 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:18.387944 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:36:18.387944 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:18.387944 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:36:18.387944 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:36:18.387944 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:18.387944 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:36:18.387944 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:18.387944 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:36:18.387944 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:36:18.387944 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:18.387944 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:36:18.388487 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:18.388487 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:36:18.389001 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:18.389001 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:36:18.389001 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:36:18.389001 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:36:18.389001 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:36:18.389001 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:36:18.389001 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:18.389001 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:36:18.389001 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:36:18.389001 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:18.389001 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:36:18.389001 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:36:18.389001 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:18.389001 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:18.389001 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:36:18.389001 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:18.390780 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:36:18.390780 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:36:18.390780 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:18.390780 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:18.390780 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:18.390780 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:36:18.390780 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:18.390780 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:18.390780 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:18.390780 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:36:18.390780 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:18.390780 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:36:18.390780 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:36:18.390780 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:18.390780 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:36:18.390780 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:18.390780 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:36:18.390780 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:36:18.390780 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:18.391737 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:36:18.391737 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:18.391737 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:36:18.391737 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:36:18.392095 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:36:18.392095 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:36:18.392207 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:36:18.392611 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:36:18.392832 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:18.393403 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:36:18.393403 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:18.393403 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:36:18.393403 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:36:18.393403 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:18.393403 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:36:18.393403 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:18.393403 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:36:18.394919 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:18.394919 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:36:18.394919 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:36:18.394919 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:36:18.394919 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:18.394919 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:36:18.394919 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:18.394919 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:36:18.394919 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:18.394919 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:36:18.394919 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:18.394919 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:36:18.394919 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:18.394919 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:36:18.394919 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:36:18.396086 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:18.396086 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:36:18.396086 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:18.396086 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:36:18.396086 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:36:18.396086 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:18.396086 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:36:18.396086 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:18.396086 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:36:18.396086 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:18.396086 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:36:18.396086 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:18.396086 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:36:18.396086 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:36:18.396953 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:36:18.396953 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:36:18.396953 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:36:18.396953 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:36:18.396953 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:36:18.396953 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:18.396953 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:18.396953 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:18.396953 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:36:18.396953 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:18.396953 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:18.396953 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:18.396953 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:36:18.397189 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:18.397189 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:36:18.397189 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:36:18.397189 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:18.397189 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:36:18.397189 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:18.397189 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:36:18.397189 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:36:18.397189 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:18.397189 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:36:18.397189 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:18.397189 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:36:18.397517 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:18.397517 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:36:18.397517 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:36:18.397517 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:36:18.397517 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:36:18.397517 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:36:18.397517 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:18.397517 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:36:18.397517 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:36:18.397517 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:18.397517 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:36:18.397517 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:36:18.397517 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:18.397517 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:18.397517 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:36:18.397517 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:18.397811 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:36:18.397811 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:36:18.397811 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:36:18.397811 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:36:18.397811 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:36:18.397811 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:18.397811 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:18.397811 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:18.397811 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:36:18.397811 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:18.397811 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:18.397811 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:18.397811 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:36:18.397811 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:18.397811 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:36:18.397811 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:36:18.397811 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:18.397811 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:36:18.398130 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:18.398130 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:36:18.398130 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:36:18.398130 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:18.398130 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:36:18.398130 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:18.398130 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:36:18.400467 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:36:18.400791 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:36:18.400907 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:36:18.401048 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:36:18.401555 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:36:18.401931 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:18.402702 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:36:18.402702 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:18.402702 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:36:18.402702 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:36:18.402702 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:18.402702 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:36:18.402702 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:18.402702 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:36:18.403533 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:18.403533 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:36:18.403533 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:36:18.403533 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:36:18.403533 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:18.403533 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:36:18.403533 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:18.403533 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:36:18.403533 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:18.403533 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:36:18.403533 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:18.403533 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:36:18.403533 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:18.403533 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:36:18.403533 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:36:18.403814 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:18.403814 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:36:18.403814 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:18.403814 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:36:18.403814 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:36:18.403814 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:18.403814 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:36:18.403814 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:18.403814 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:36:18.404468 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:18.404468 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:36:18.404468 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:18.404468 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:36:18.404468 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:36:18.404468 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:36:18.404468 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:36:18.404468 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:36:18.404468 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:36:18.404468 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:18.404468 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:18.404712 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:18.404712 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:36:18.404712 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:18.404712 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:18.404712 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:18.404712 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:36:18.404712 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:18.404712 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:36:18.404712 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:36:18.404712 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:18.404712 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:36:18.404712 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:18.404712 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:36:18.404712 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:36:18.405114 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:18.405114 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:36:18.405114 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:18.405114 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:36:18.405551 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:18.405551 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:36:18.405551 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:36:18.405551 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:36:18.405551 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:36:18.405551 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:36:18.405551 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:18.405551 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:36:18.405551 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:36:18.405551 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:18.405551 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:36:18.405551 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:36:18.405551 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:18.405551 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:18.405551 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:36:18.405551 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:18.405551 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:36:18.406120 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:36:18.406120 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:36:18.406120 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:36:18.406120 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:18.406120 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:18.406120 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:18.406120 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:36:18.406120 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:18.406120 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:18.406120 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:18.406120 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:36:18.406120 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:18.406120 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:36:18.406120 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:36:18.406120 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:18.406120 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:36:18.406120 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:18.406120 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:36:18.406120 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:36:18.406741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:18.406741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:36:18.406741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:18.406741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:36:18.410432 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:36:18.410752 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:36:18.410874 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:36:18.411051 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:36:18.411529 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:36:18.411847 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:36:18.412054 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:18.412776 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:36:18.412776 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:18.412776 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:36:18.412776 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:36:18.412776 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:18.412776 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:36:18.412776 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:18.412776 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:36:18.413705 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:18.413705 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:36:18.413705 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:18.413705 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:36:18.413705 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:36:18.413705 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:36:18.413705 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:36:18.413705 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:18.413705 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:18.413705 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:18.413705 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:36:18.413705 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:18.414065 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:18.414065 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:18.414065 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:36:18.414065 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:18.414065 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:36:18.414065 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:36:18.414065 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:18.414065 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:36:18.414065 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:18.414065 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:36:18.414065 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:36:18.414065 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:18.414065 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:36:18.414423 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:18.414423 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:36:18.414734 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:18.414734 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:36:18.414734 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:36:18.414734 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:36:18.414734 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:36:18.414734 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:36:18.414734 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:18.414734 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:36:18.414734 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:36:18.414734 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:18.414734 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:36:18.414734 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:36:18.414734 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:18.414734 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:18.414734 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:36:18.414734 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:18.415239 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:36:18.415239 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:36:18.415239 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:18.415239 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:18.415239 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:18.415239 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:36:18.415239 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:18.415239 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:18.415239 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:18.415239 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:36:18.415239 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:18.415239 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:36:18.415239 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:36:18.415239 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:18.415239 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:36:18.415239 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:18.415239 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:36:18.415239 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:36:18.415239 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:18.415754 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:36:18.415754 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:18.415754 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:36:22.918069 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:36:22.918974 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:36:22.918974 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:36:22.920040 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:36:22.951952 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:36:22.952287 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:22.953049 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:36:22.953049 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:22.953049 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:36:22.953049 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:36:22.953049 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:22.953049 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:36:22.953049 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:22.953049 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:36:22.953724 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.034 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:22.953724 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:36:22.953724 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:22.953724 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:36:22.953724 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:36:22.953724 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:36:22.953724 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:36:22.953724 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:22.953724 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:22.953724 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:22.953724 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:36:22.953724 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:22.954661 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:22.954661 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:22.954661 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:36:22.954661 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:22.954661 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:36:22.954661 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:36:22.954661 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:22.954661 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:36:22.954661 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:22.954661 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:36:22.954661 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:36:22.954661 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:22.954661 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:36:22.955523 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:22.955523 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:36:22.955523 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:22.955523 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:36:22.955523 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:36:22.955523 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:36:22.955523 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:36:22.955523 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:36:22.955523 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:22.955523 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:36:22.955523 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:36:22.955523 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:22.955523 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:36:22.955523 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:36:22.955523 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:22.956509 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:22.956509 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:36:22.956509 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:22.956509 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:36:22.956509 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:36:22.956509 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:22.956509 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:22.956509 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:22.956509 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:36:22.956509 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:22.956509 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:22.956509 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:22.956509 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:36:22.956509 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:22.956509 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:36:22.956509 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:36:22.956509 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:22.956509 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:36:22.956509 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:22.957742 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:36:22.957742 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:36:22.957742 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:22.957742 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:36:22.957742 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:22.957742 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:36:23.382482 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:36:23.383201 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:36:23.383437 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:36:23.384578 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:36:23.384987 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:23.386050 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:36:23.386050 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:23.386050 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:36:23.386050 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:36:23.386050 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:23.386050 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:36:23.386050 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:23.386050 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:36:23.387699 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:23.387699 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:36:23.387699 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:23.387699 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:36:23.387699 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:36:23.387699 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:36:23.387699 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:36:23.387699 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:23.387699 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:23.387699 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:23.387699 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:36:23.387699 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:23.388243 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:23.388243 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:23.388243 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:36:23.388243 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:23.388243 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:36:23.388243 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:36:23.388243 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:23.388243 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:36:23.388243 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:23.388243 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:36:23.388243 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:36:23.388243 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:23.388243 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:36:23.388799 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:23.388799 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:36:23.389458 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:23.389458 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:36:23.389458 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:36:23.389458 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:36:23.389458 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:36:23.389458 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:36:23.389458 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:23.389458 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:36:23.389458 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:36:23.389458 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:23.389458 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:36:23.389458 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:36:23.389458 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:23.389458 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:23.389458 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:36:23.389458 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:23.390158 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:36:23.390158 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:36:23.390158 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:23.390158 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:23.390158 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:23.390158 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:36:23.390158 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:23.390158 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:23.390158 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:23.390158 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:36:23.390158 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:23.390158 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:36:23.390158 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:36:23.390158 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:23.390158 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:36:23.390158 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:23.390158 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:36:23.390158 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:36:23.390158 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:23.390931 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:36:23.390931 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:23.390931 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:36:23.392135 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:36:23.392480 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:36:23.392621 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:36:23.392784 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:36:23.393344 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:36:23.393698 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:23.394346 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:36:23.394346 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:23.394346 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:36:23.394346 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:36:23.394346 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:23.394346 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:36:23.394346 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:23.394346 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:36:23.395198 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:23.395198 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:36:23.395198 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:36:23.395198 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:36:23.395198 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:23.395198 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:36:23.395198 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:23.395198 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:36:23.395198 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:23.395198 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:36:23.395198 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:23.395198 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:36:23.395198 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:23.395198 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:36:23.395198 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:36:23.395925 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:23.395925 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:36:23.395925 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:23.395925 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:36:23.395925 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:36:23.395925 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:23.395925 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:36:23.395925 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:23.395925 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:36:23.396371 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:23.396371 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:36:23.396371 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:23.396371 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:36:23.396371 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:36:23.396371 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:36:23.396371 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:36:23.396371 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:36:23.396371 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:36:23.396371 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:36:23.396371 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:23.396371 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:23.396940 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:23.396940 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:36:23.396940 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:23.396940 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:23.396940 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:23.396940 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:36:23.396940 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:23.396940 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:36:23.396940 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:36:23.396940 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:23.396940 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:36:23.396940 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:23.396940 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:36:23.396940 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:36:23.397599 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:23.397599 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:36:23.397599 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:23.397599 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:36:23.397837 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:23.397837 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:36:23.397837 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:36:23.397837 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:36:23.397837 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:36:23.397837 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:36:23.397837 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:23.397837 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:36:23.397837 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:36:23.397837 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:23.397837 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:36:23.397837 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:36:23.397837 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:23.397837 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:23.397837 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:36:23.397837 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:23.398285 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:36:23.398285 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:36:23.398285 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:36:23.398285 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:36:23.398285 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:36:23.398285 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:23.398285 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:23.398285 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:23.398285 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:36:23.398285 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:23.398285 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:23.398285 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:23.398285 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:36:23.398285 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:23.398285 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:36:23.398285 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:36:23.398285 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:23.398285 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:36:23.398789 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:23.398789 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:36:23.398789 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:36:23.398789 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:23.398789 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:36:23.398789 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:23.398789 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:36:23.400773 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:36:23.401003 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:36:23.401137 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:36:23.401299 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:36:23.401581 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:36:23.401819 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:23.402250 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:36:23.402250 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:23.402250 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:36:23.402250 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:36:23.402250 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:23.402250 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:36:23.402250 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:23.402250 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:36:23.403041 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:23.403041 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:36:23.403041 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:36:23.403041 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:36:23.403041 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:23.403041 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:36:23.403041 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:23.403041 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:36:23.403041 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:23.403041 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:36:23.403041 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:23.403041 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:36:23.403041 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:23.403041 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:36:23.403041 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:36:23.403425 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:23.403425 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:36:23.403425 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:23.403425 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:36:23.403425 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:36:23.403425 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:23.403425 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:36:23.403425 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:23.403425 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:36:23.404075 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:23.404075 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:36:23.404075 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:23.404075 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:36:23.404075 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:36:23.404075 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:36:23.404075 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:36:23.404075 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:36:23.404075 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:36:23.404075 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:23.404075 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:23.404391 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:23.404391 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:36:23.404391 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:23.404391 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:23.404391 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:23.404391 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:36:23.404391 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:23.404391 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:36:23.404391 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:36:23.404391 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:23.404391 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:36:23.404391 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:23.404391 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:36:23.404391 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:36:23.404738 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:23.404738 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:36:23.404738 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:23.404738 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:36:23.405340 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:23.405340 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:36:23.405340 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:36:23.405340 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:36:23.405340 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:36:23.405340 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:36:23.405340 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:23.405340 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:36:23.405340 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:36:23.405340 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:23.405340 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:36:23.405340 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:36:23.405340 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:23.405340 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:23.405340 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:36:23.405340 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:23.405340 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:36:23.405836 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:36:23.405836 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:36:23.405836 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:36:23.405836 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:23.405836 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:23.405836 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:23.405836 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:36:23.405836 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:23.405836 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:23.405836 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:23.405836 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:36:23.405836 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:23.405836 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:36:23.405836 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:36:23.405836 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:23.405836 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:36:23.405836 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:23.405836 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:36:23.405836 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:36:23.406296 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:23.406296 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:36:23.406296 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:23.406296 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:36:23.410844 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:36:23.411097 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:36:23.411232 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:36:23.411461 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:36:23.411809 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:36:23.412150 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:36:23.412333 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:23.413020 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:36:23.413020 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:23.413020 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:36:23.413020 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:36:23.413020 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:23.413020 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:36:23.413020 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:23.413020 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:36:23.413910 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:23.413910 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:36:23.413910 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:23.413910 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:36:23.413910 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:36:23.413910 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:36:23.413910 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:36:23.413910 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:23.413910 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:23.413910 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:23.413910 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:36:23.413910 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:23.414264 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:23.414264 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:23.414264 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:36:23.414264 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:23.414264 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:36:23.414264 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:36:23.414264 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:23.414264 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:36:23.414264 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:23.414264 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:36:23.414264 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:36:23.414264 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:23.414264 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:36:23.414622 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:23.414622 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:36:23.415103 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:23.415103 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:36:23.415103 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:36:23.415103 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:36:23.415103 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:36:23.415103 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:36:23.415103 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:23.415103 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:36:23.415103 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:36:23.415103 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:23.415103 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:36:23.415103 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:36:23.415103 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:23.415103 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:23.415103 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:36:23.415103 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:23.415858 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:36:23.415858 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:36:23.415858 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:23.415858 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:23.415858 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:23.415858 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:36:23.415858 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:23.415858 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:23.415858 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:23.415858 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:36:23.415858 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:23.415858 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:36:23.415858 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:36:23.415858 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:23.415858 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:36:23.415858 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:23.415858 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:36:23.415858 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:36:23.415858 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:23.416731 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:36:23.416731 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:23.416731 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:36:27.918717 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:36:27.919621 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:36:27.919621 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:36:27.920529 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:36:27.956515 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:36:27.956989 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:27.957933 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:36:27.957933 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:27.957933 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:36:27.957933 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:36:27.957933 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:27.957933 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:36:27.957933 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:27.957933 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:36:27.958801 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.038 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:27.958801 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:36:27.958801 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:27.958801 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:36:27.958801 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:36:27.958801 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:36:27.958801 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:36:27.958801 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:27.958801 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:27.958801 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:27.958801 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:36:27.958801 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:27.959183 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:27.959183 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:27.959183 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:36:27.959183 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:27.959183 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:36:27.959183 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:36:27.959183 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:27.959183 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:36:27.959183 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:27.959183 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:36:27.959183 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:36:27.959183 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:27.959183 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:36:27.959662 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:27.959662 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:36:27.959662 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:27.959662 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:36:27.959662 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:36:27.959662 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:36:27.959662 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:36:27.959662 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:36:27.959662 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:27.959662 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:36:27.959662 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:36:27.959662 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:27.959662 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:36:27.959662 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:36:27.959662 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:27.960105 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:27.960105 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:36:27.960105 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:27.960105 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:36:27.960105 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:36:27.960105 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:27.960105 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:27.960105 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:27.960105 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:36:27.960105 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:27.960105 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:27.960105 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:27.960105 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:36:27.960105 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:27.960105 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:36:27.960105 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:36:27.960105 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:27.960105 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:36:27.960105 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:27.960933 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:36:27.960933 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:36:27.960933 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:27.960933 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:36:27.960933 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:27.960933 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:36:28.383522 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:36:28.384186 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:36:28.384446 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:36:28.385799 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:36:28.386195 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:28.387166 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:36:28.387166 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:28.387166 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:36:28.387166 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:36:28.387166 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:28.387166 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:36:28.387166 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:28.387166 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:36:28.388756 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:28.388756 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:36:28.388756 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:28.388756 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:36:28.388756 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:36:28.388756 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:36:28.388756 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:36:28.388756 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:28.388756 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:28.388756 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:28.388756 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:36:28.388756 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:28.389449 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:28.389449 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:28.389449 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:36:28.389449 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:28.389449 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:36:28.389449 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:36:28.389449 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:28.389449 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:36:28.389449 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:28.389449 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:36:28.389449 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:36:28.389449 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:28.389449 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:36:28.389772 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:28.389772 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:36:28.390460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:28.390460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:36:28.390460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:36:28.390460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:36:28.390460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:36:28.390460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:36:28.390460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:28.390460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:36:28.390460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:36:28.390460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:28.390460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:36:28.390460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:36:28.390460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:28.390460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:28.390460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:36:28.390460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:28.390861 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:36:28.390861 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:36:28.390861 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:28.390861 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:28.390861 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:28.390861 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:36:28.390861 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:28.390861 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:28.390861 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:28.390861 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:36:28.390861 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:28.390861 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:36:28.390861 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:36:28.390861 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:28.390861 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:36:28.390861 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:28.390861 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:36:28.390861 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:36:28.390861 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:28.391323 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:36:28.391323 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:28.391323 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:36:28.392665 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:36:28.393054 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:36:28.393214 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:36:28.393400 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:36:28.393929 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:36:28.394248 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:28.394992 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:36:28.394992 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:28.394992 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:36:28.394992 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:36:28.394992 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:28.394992 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:36:28.394992 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:28.394992 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:36:28.395816 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:28.395816 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:36:28.395816 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:36:28.395816 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:36:28.395816 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:28.395816 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:36:28.395816 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:28.395816 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:36:28.395816 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:28.395816 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:36:28.395816 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:28.395816 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:36:28.395816 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:28.395816 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:36:28.395816 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:36:28.396200 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:28.396200 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:36:28.396200 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:28.396200 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:36:28.396200 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:36:28.396200 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:28.396200 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:36:28.396200 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:28.396200 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:36:28.396693 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:28.396693 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:36:28.396693 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:28.396693 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:36:28.396693 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:36:28.396693 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:36:28.396693 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:36:28.396693 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:36:28.396693 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:36:28.396693 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:36:28.396693 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:28.396693 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:28.397628 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:28.397628 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:36:28.397628 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:28.397628 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:28.397628 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:28.397628 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:36:28.397628 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:28.397628 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:36:28.397628 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:36:28.397628 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:28.397628 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:36:28.397628 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:28.397628 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:36:28.397628 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:36:28.398109 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:28.398109 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:36:28.398109 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:28.398109 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:36:28.398109 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:28.398109 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:36:28.398109 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:36:28.398109 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:36:28.398109 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:36:28.398109 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:36:28.398109 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:28.398109 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:36:28.398109 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:36:28.398109 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:28.398460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:36:28.398460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:36:28.398460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:28.398460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:28.398460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:36:28.398460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:28.398460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:36:28.398460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:36:28.398460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:36:28.398460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:36:28.398460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:36:28.398460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:28.398460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:28.398460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:28.398460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:36:28.398460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:28.398460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:28.398460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:28.398460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:36:28.398917 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:28.398917 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:36:28.398917 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:36:28.398917 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:28.398917 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:36:28.398917 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:28.398917 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:36:28.398917 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:36:28.398917 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:28.398917 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:36:28.398917 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:28.398917 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:36:28.401298 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:36:28.401666 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:36:28.401777 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:36:28.401912 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:36:28.402511 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:36:28.402898 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:28.403669 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:36:28.403669 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:28.403669 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:36:28.403669 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:36:28.403669 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:28.403669 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:36:28.403669 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:28.403669 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:36:28.404469 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:28.404469 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:36:28.404469 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:36:28.404469 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:36:28.404469 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:28.404469 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:36:28.404469 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:28.404469 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:36:28.404469 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:28.404469 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:36:28.404469 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:28.404469 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:36:28.404469 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:28.404469 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:36:28.404469 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:36:28.404841 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:28.404841 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:36:28.404841 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:28.404841 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:36:28.404841 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:36:28.404841 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:28.404841 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:36:28.404841 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:28.404841 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:36:28.405392 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:28.405392 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:36:28.405392 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:28.405392 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:36:28.405392 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:36:28.405392 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:36:28.405392 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:36:28.405392 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:36:28.405392 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:36:28.405392 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:28.405392 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:28.406162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:28.406162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:36:28.406162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:28.406162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:28.406162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:28.406162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:36:28.406162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:28.406162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:36:28.406162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:36:28.406162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:28.406162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:36:28.406162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:28.406162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:36:28.406162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:36:28.406742 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:28.406742 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:36:28.406742 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:28.406742 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:36:28.406742 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:28.406742 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:36:28.406742 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:36:28.406742 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:36:28.406742 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:36:28.406742 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:36:28.406742 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:28.406742 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:36:28.406742 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:36:28.406742 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:28.407080 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:36:28.407080 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:36:28.407080 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:28.407080 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:28.407080 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:36:28.407080 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:28.407080 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:36:28.407080 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:36:28.407080 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:36:28.407080 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:36:28.407080 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:28.407080 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:28.407080 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:28.407080 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:36:28.407080 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:28.407080 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:28.407080 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:28.407080 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:36:28.407080 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:28.407551 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:36:28.407551 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:36:28.407551 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:28.407551 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:36:28.407551 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:28.407551 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:36:28.407551 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:36:28.407551 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:28.407551 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:36:28.407551 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:28.407551 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:36:28.411391 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:36:28.411732 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:36:28.411840 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:36:28.412006 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:36:28.412544 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:36:28.412913 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:36:28.413118 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:28.413930 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:36:28.413930 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:28.413930 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:36:28.413930 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:36:28.413930 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:28.413930 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:36:28.413930 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:28.413930 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:36:28.414809 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:28.414809 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:36:28.414809 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:28.414809 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:36:28.414809 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:36:28.414809 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:36:28.414809 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:36:28.414809 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:28.414809 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:28.414809 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:28.414809 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:36:28.414809 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:28.415146 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:28.415146 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:28.415146 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:36:28.415146 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:28.415146 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:36:28.415146 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:36:28.415146 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:28.415146 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:36:28.415146 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:28.415146 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:36:28.415146 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:36:28.415146 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:28.415146 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:36:28.415535 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:28.415535 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:36:28.415761 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:28.415761 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:36:28.415761 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:36:28.415761 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:36:28.415761 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:36:28.415761 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:36:28.415761 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:28.415761 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:36:28.415761 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:36:28.415761 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:28.415761 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:36:28.415761 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:36:28.415761 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:28.415761 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:28.415761 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:36:28.415761 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:28.416220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:36:28.416220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:36:28.416220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:28.416220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:28.416220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:28.416220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:36:28.416220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:28.416220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:28.416220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:28.416220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:36:28.416220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:28.416220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:36:28.416220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:36:28.416220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:28.416220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:36:28.416220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:28.416220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:36:28.416220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:36:28.416220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:28.416774 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:36:28.416774 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:28.416774 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:36:32.919977 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:36:32.920914 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:36:32.920914 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:36:32.922252 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:36:32.962276 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:36:32.962796 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:32.963843 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:36:32.963843 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:32.963843 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:36:32.963843 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:36:32.963843 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:32.963843 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:36:32.963843 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:32.963843 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:36:32.964719 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.043 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:32.964719 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:36:32.964719 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:32.964719 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:36:32.964719 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:36:32.964719 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:36:32.964719 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:36:32.964719 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:32.964719 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:32.964719 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:32.964719 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:36:32.964719 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:32.965428 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:32.965428 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:32.965428 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:36:32.965428 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:32.965428 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:36:32.965428 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:36:32.965428 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:32.965428 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:36:32.965428 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:32.965428 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:36:32.965428 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:36:32.965428 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:32.965428 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:36:32.966057 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:32.966057 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:36:32.966057 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:32.966057 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:36:32.966057 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:36:32.966057 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:36:32.966057 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:36:32.966057 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:36:32.966057 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:32.966057 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:36:32.966057 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:36:32.966057 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:32.966057 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:36:32.966057 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:36:32.966057 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:32.966802 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:32.966802 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:36:32.966802 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:32.966802 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:36:32.966802 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:36:32.966802 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:32.966802 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:32.966802 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:32.966802 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:36:32.966802 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:32.966802 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:32.966802 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:32.966802 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:36:32.966802 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:32.966802 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:36:32.966802 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:36:32.966802 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:32.966802 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:36:32.966802 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:32.967660 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:36:32.967660 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:36:32.967660 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:32.967660 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:36:32.967660 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:32.967660 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:36:33.384517 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:36:33.385298 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:36:33.385524 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:36:33.386931 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:36:33.387414 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:33.388506 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:36:33.388506 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:33.388506 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:36:33.388506 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:36:33.388506 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:33.388506 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:36:33.388506 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:33.388506 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:36:33.390197 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:33.390197 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:36:33.390197 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:33.390197 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:36:33.390197 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:36:33.390197 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:36:33.390197 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:36:33.390197 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:33.390197 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:33.390197 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:33.390197 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:36:33.390197 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:33.390870 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:33.390870 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:33.390870 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:36:33.390870 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:33.390870 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:36:33.390870 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:36:33.390870 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:33.390870 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:36:33.390870 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:33.390870 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:36:33.390870 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:36:33.390870 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:33.390870 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:36:33.391476 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:33.391476 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:36:33.392056 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:33.392056 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:36:33.392056 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:36:33.392056 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:36:33.392056 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:36:33.392056 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:36:33.392056 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:33.392056 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:36:33.392056 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:36:33.392056 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:33.392056 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:36:33.392056 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:36:33.392056 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:33.392056 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:33.392056 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:36:33.392056 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:33.392820 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:36:33.392820 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:36:33.392820 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:33.392820 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:33.392820 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:33.392820 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:36:33.392820 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:33.392820 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:33.392820 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:33.392820 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:36:33.392820 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:33.392820 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:36:33.392820 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:36:33.392820 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:33.392820 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:36:33.392820 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:33.392820 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:36:33.392820 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:36:33.392820 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:33.393649 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:36:33.393649 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:33.393649 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:36:33.393649 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:36:33.393649 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:36:33.393910 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:36:33.393910 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:36:33.394517 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:36:33.394896 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:33.395727 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:36:33.395727 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:33.395727 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:36:33.395727 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:36:33.395727 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:33.395727 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:36:33.395727 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:33.395727 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:36:33.396598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:33.396598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:36:33.396598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:36:33.396598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:36:33.396598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:33.396598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:36:33.396598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:33.396598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:36:33.396598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:33.396598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:36:33.396598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:33.396598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:36:33.396598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:33.396598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:36:33.396598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:36:33.397052 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:33.397052 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:36:33.397052 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:33.397052 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:36:33.397052 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:36:33.397052 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:33.397052 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:36:33.397052 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:33.397052 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:36:33.397617 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:33.397617 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:36:33.397617 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:33.397617 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:36:33.397617 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:36:33.397617 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:36:33.397617 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:36:33.397617 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:36:33.397617 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:36:33.397617 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:36:33.397617 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:33.397617 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:33.397982 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:33.397982 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:36:33.397982 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:33.397982 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:33.397982 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:33.397982 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:36:33.397982 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:33.397982 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:36:33.397982 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:36:33.397982 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:33.397982 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:36:33.397982 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:33.397982 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:36:33.397982 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:36:33.398403 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:33.398403 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:36:33.398403 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:33.398403 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:36:33.398802 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:33.398802 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:36:33.398802 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:36:33.398802 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:36:33.398802 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:36:33.398802 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:36:33.398802 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:33.398802 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:36:33.398802 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:36:33.398802 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:33.398802 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:36:33.398802 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:36:33.398802 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:33.398802 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:33.398802 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:36:33.398802 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:33.399241 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:36:33.399241 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:36:33.399241 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:36:33.399241 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:36:33.399241 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:36:33.399241 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:33.399241 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:33.399241 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:33.399241 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:36:33.399241 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:33.399241 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:33.399241 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:33.399241 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:36:33.399241 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:33.399241 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:36:33.399241 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:36:33.399241 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:33.399241 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:36:33.399842 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:33.399842 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:36:33.399842 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:36:33.399842 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:33.399842 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:36:33.399842 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:33.399842 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:36:33.401848 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:36:33.402171 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:36:33.402287 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:36:33.402447 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:36:33.402860 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:36:33.403095 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:33.403682 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:36:33.403682 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:33.403682 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:36:33.403682 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:36:33.403682 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:33.403682 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:36:33.403682 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:33.403682 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:36:33.404459 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:33.404459 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:36:33.404459 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:36:33.404459 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:36:33.404459 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:33.404459 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:36:33.404459 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:33.404459 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:36:33.404459 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:33.404459 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:36:33.404459 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:33.404459 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:36:33.404459 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:33.404459 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:36:33.404459 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:36:33.404871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:33.404871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:36:33.404871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:33.404871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:36:33.404871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:36:33.404871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:33.404871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:36:33.404871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:33.404871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:36:33.405302 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:33.405302 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:36:33.405302 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:33.405302 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:36:33.405302 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:36:33.405302 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:36:33.405302 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:36:33.405302 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:36:33.405302 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:36:33.405302 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:33.405302 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:33.405679 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:33.405679 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:36:33.405679 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:33.405679 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:33.405679 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:33.405679 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:36:33.405679 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:33.405679 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:36:33.405679 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:36:33.405679 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:33.405679 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:36:33.405679 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:33.405679 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:36:33.405679 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:36:33.406071 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:33.406071 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:36:33.406071 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:33.406071 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:36:33.406353 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:33.406353 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:36:33.406353 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:36:33.406353 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:36:33.406353 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:36:33.406353 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:36:33.406353 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:33.406353 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:36:33.406353 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:36:33.406353 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:33.406353 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:36:33.406353 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:36:33.406353 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:33.406353 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:33.406353 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:36:33.406353 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:33.406353 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:36:33.406863 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:36:33.406863 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:36:33.406863 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:36:33.406863 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:33.406863 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:33.406863 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:33.406863 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:36:33.406863 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:33.406863 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:33.406863 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:33.406863 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:36:33.406863 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:33.406863 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:36:33.406863 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:36:33.406863 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:33.406863 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:36:33.406863 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:33.406863 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:36:33.406863 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:36:33.407398 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:33.407398 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:36:33.407398 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:33.407398 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:36:33.411970 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:36:33.412304 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:36:33.412410 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:36:33.412585 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:36:33.413060 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:36:33.413485 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:36:33.413681 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:33.414511 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:36:33.414511 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:33.414511 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:36:33.414511 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:36:33.414511 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:33.414511 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:36:33.414511 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:33.414511 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:36:33.415401 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:33.415401 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:36:33.415401 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:33.415401 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:36:33.415401 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:36:33.415401 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:36:33.415401 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:36:33.415401 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:33.415401 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:33.415401 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:33.415401 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:36:33.415401 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:33.415756 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:33.415756 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:33.415756 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:36:33.415756 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:33.415756 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:36:33.415756 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:36:33.415756 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:33.415756 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:36:33.415756 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:33.415756 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:36:33.415756 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:36:33.415756 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:33.415756 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:36:33.416111 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:33.416111 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:36:33.416353 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:33.416353 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:36:33.416353 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:36:33.416353 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:36:33.416353 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:36:33.416353 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:36:33.416353 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:33.416353 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:36:33.416353 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:36:33.416353 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:33.416353 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:36:33.416353 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:36:33.416353 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:33.416353 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:33.416353 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:36:33.416353 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:33.416790 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:36:33.416790 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:36:33.416790 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:33.416790 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:33.416790 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:33.416790 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:36:33.416790 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:33.416790 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:33.416790 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:33.416790 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:36:33.416790 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:33.416790 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:36:33.416790 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:36:33.416790 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:33.416790 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:36:33.416790 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:33.416790 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:36:33.416790 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:36:33.416790 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:33.417306 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:36:33.417306 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:33.417306 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:36:37.921422 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:36:37.922280 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:36:37.922468 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:36:37.923932 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:36:37.957197 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:36:37.957691 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:37.958785 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:36:37.958785 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:37.958785 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:36:37.958785 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:36:37.958785 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:37.958785 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:36:37.958785 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:37.958785 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:36:37.959771 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.036 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:37.959771 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:36:37.959771 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:37.959771 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:36:37.959771 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:36:37.959771 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:36:37.959771 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:36:37.959771 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:37.959771 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:37.959771 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:37.959771 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:36:37.959771 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:37.960909 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:37.960909 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:37.960909 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:36:37.960909 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:37.960909 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:36:37.960909 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:36:37.960909 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:37.960909 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:36:37.960909 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:37.960909 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:36:37.960909 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:36:37.960909 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:37.960909 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:36:37.961486 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:37.961486 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:36:37.961486 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:37.961486 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:36:37.961486 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:36:37.961486 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:36:37.961486 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:36:37.961486 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:36:37.961486 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:37.961486 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:36:37.961486 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:36:37.961486 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:37.961486 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:36:37.961486 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:36:37.961486 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:37.963041 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:37.963041 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:36:37.963041 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:37.963041 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:36:37.963041 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:36:37.963041 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:37.963041 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:37.963041 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:37.963041 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:36:37.963041 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:37.963041 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:37.963041 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:37.963041 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:36:37.963041 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:37.963041 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:36:37.963041 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:36:37.963041 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:37.963041 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:36:37.963041 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:37.964294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:36:37.964294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:36:37.964294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:37.964294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:36:37.964294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:37.964294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:36:38.385516 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:36:38.386346 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:36:38.386550 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:36:38.387787 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:36:38.388207 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:38.389276 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:36:38.389276 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:38.389276 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:36:38.389276 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:36:38.389276 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:38.389276 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:36:38.389276 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:38.389276 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:36:38.391032 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:38.391032 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:36:38.391032 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:38.391032 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:36:38.391032 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:36:38.391032 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:36:38.391032 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:36:38.391032 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:38.391032 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:38.391032 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:38.391032 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:36:38.391032 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:38.391389 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:38.391389 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:38.391389 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:36:38.391389 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:38.391389 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:36:38.391389 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:36:38.391389 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:38.391389 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:36:38.391389 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:38.391389 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:36:38.391389 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:36:38.391389 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:38.391389 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:36:38.391730 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:38.391730 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:36:38.393147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:38.393147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:36:38.393147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:36:38.393147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:36:38.393147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:36:38.393147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:36:38.393147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:38.393147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:36:38.393147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:36:38.393147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:38.393147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:36:38.393147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:36:38.393147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:38.393147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:38.393147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:36:38.393147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:38.393589 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:36:38.393589 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:36:38.393589 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:38.393589 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:38.393589 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:38.393589 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:36:38.393589 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:38.393589 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:38.393589 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:38.393589 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:36:38.393589 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:38.393589 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:36:38.393589 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:36:38.393589 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:38.393589 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:36:38.393589 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:38.393589 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:36:38.393589 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:36:38.393589 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:38.394101 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:36:38.394101 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:38.394101 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:36:38.394101 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:36:38.394101 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:36:38.394369 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:36:38.394439 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:36:38.394967 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:36:38.395284 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:38.396076 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:36:38.396076 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:38.396076 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:36:38.396076 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:36:38.396076 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:38.396076 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:36:38.396076 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:38.396076 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:36:38.396801 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:38.396801 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:36:38.396801 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:36:38.396801 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:36:38.396801 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:38.396801 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:36:38.396801 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:38.396801 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:36:38.396801 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:38.396801 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:36:38.396801 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:38.396801 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:36:38.396801 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:38.396801 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:36:38.396801 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:36:38.397213 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:38.397213 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:36:38.397213 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:38.397213 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:36:38.397213 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:36:38.397213 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:38.397213 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:36:38.397213 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:38.397213 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:36:38.397639 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:38.397639 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:36:38.397639 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:38.397639 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:36:38.397639 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:36:38.397639 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:36:38.397639 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:36:38.397639 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:36:38.397639 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:36:38.397639 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:36:38.397639 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:38.397639 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:38.398045 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:38.398045 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:36:38.398045 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:38.398045 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:38.398045 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:38.398045 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:36:38.398045 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:38.398045 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:36:38.398045 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:36:38.398045 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:38.398045 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:36:38.398045 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:38.398045 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:36:38.398045 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:36:38.398558 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:38.398558 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:36:38.398558 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:38.398558 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:36:38.398773 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:38.398773 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:36:38.398773 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:36:38.398773 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:36:38.398773 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:36:38.398773 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:36:38.398773 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:38.398773 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:36:38.398773 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:36:38.398773 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:38.398773 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:36:38.398773 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:36:38.398773 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:38.398773 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:38.398773 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:36:38.398773 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:38.399586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:36:38.399586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:36:38.399586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:36:38.399586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:36:38.399586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:36:38.399586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:38.399586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:38.399586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:38.399586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:36:38.399586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:38.399586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:38.399586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:38.399586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:36:38.399586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:38.399586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:36:38.399586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:36:38.399586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:38.399586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:36:38.400037 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:38.400037 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:36:38.400037 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:36:38.400037 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:38.400037 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:36:38.400037 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:38.400037 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:36:38.402343 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:36:38.402709 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:36:38.402815 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:36:38.402951 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:36:38.403471 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:36:38.403918 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:38.404595 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:36:38.404595 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:38.404595 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:36:38.404595 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:36:38.404595 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:38.404595 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:36:38.404595 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:38.404595 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:36:38.405346 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:38.405346 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:36:38.405346 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:36:38.405346 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:36:38.405346 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:38.405346 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:36:38.405346 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:38.405346 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:36:38.405346 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:38.405346 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:36:38.405346 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:38.405346 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:36:38.405346 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:38.405346 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:36:38.405346 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:36:38.405835 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:38.405835 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:36:38.405835 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:38.405835 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:36:38.405835 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:36:38.405835 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:38.405835 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:36:38.405835 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:38.405835 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:36:38.406217 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:38.406217 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:36:38.406217 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:38.406217 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:36:38.406217 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:36:38.406217 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:36:38.406217 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:36:38.406217 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:36:38.406217 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:36:38.406217 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:38.406217 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:38.406832 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:38.406832 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:36:38.406832 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:38.406832 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:38.406832 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:38.406832 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:36:38.406832 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:38.406832 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:36:38.406832 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:36:38.406832 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:38.406832 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:36:38.406832 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:38.406832 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:36:38.406832 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:36:38.407612 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:38.407612 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:36:38.407612 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:38.407612 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:36:38.407612 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:38.407612 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:36:38.407612 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:36:38.407612 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:36:38.407612 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:36:38.407612 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:36:38.407612 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:38.407612 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:36:38.407612 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:36:38.407612 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:38.407958 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:36:38.407958 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:36:38.407958 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:38.407958 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:38.407958 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:36:38.407958 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:38.407958 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:36:38.407958 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:36:38.407958 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:36:38.407958 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:36:38.407958 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:38.407958 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:38.407958 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:38.407958 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:36:38.407958 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:38.407958 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:38.407958 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:38.407958 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:36:38.407958 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:38.408457 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:36:38.408457 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:36:38.408457 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:38.408457 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:36:38.408457 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:38.408457 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:36:38.408457 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:36:38.408457 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:38.408457 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:36:38.408457 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:38.408457 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:36:38.412603 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:36:38.412929 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:36:38.413041 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:36:38.413192 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:36:38.413556 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:36:38.413786 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:36:38.413902 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:38.414598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:36:38.414598 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:38.414598 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:36:38.414598 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:36:38.414598 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:38.414598 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:36:38.414598 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:38.414598 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:36:38.416077 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:38.416077 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:36:38.416077 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:38.416077 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:36:38.416077 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:36:38.416077 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:36:38.416077 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:36:38.416077 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:38.416077 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:38.416077 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:38.416077 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:36:38.416077 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:38.416411 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:38.416411 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:38.416411 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:36:38.416411 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:38.416411 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:36:38.416411 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:36:38.416411 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:38.416411 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:36:38.416411 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:38.416411 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:36:38.416411 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:36:38.416411 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:38.416411 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:36:38.416705 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:38.416705 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:36:38.417784 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:38.417784 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:36:38.417784 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:36:38.417784 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:36:38.417784 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:36:38.417784 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:36:38.417784 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:38.417784 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:36:38.417784 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:36:38.417784 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:38.417784 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:36:38.417784 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:36:38.417784 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:38.417784 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:38.417784 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:36:38.417784 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:38.418229 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:36:38.418229 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:36:38.418229 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:38.418229 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:38.418229 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:38.418229 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:36:38.418229 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:38.418229 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:38.418229 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:38.418229 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:36:38.418229 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:38.418229 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:36:38.418229 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:36:38.418229 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:38.418229 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:36:38.418229 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:38.418229 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:36:38.418229 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:36:38.418229 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:38.418711 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:36:38.418711 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:38.418711 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:36:42.922512 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:36:42.923223 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:36:42.923223 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:36:42.924563 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:36:42.957253 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:36:42.957704 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:42.958639 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:36:42.958639 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:42.958639 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:36:42.958639 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:36:42.958639 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:42.958639 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:36:42.958639 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:42.958639 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:36:42.959648 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.035 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:42.959648 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:36:42.959648 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:42.959648 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:36:42.959648 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:36:42.959648 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:36:42.959648 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:36:42.959648 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:42.959648 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:42.959648 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:42.959648 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:36:42.959648 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:42.960357 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:42.960357 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:42.960357 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:36:42.960357 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:42.960357 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:36:42.960357 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:36:42.960357 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:42.960357 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:36:42.960357 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:42.960357 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:36:42.960357 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:36:42.960357 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:42.960357 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:36:42.960864 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:42.960864 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:36:42.960864 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:42.960864 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:36:42.960864 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:36:42.960864 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:36:42.960864 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:36:42.960864 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:36:42.960864 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:42.960864 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:36:42.960864 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:36:42.960864 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:42.960864 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:36:42.960864 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:36:42.960864 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:42.961266 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:42.961266 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:36:42.961266 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:42.961266 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:36:42.961266 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:36:42.961266 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:42.961266 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:42.961266 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:42.961266 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:36:42.961266 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:42.961266 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:42.961266 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:42.961266 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:36:42.961266 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:42.961266 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:36:42.961266 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:36:42.961266 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:42.961266 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:36:42.961266 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:42.961849 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:36:42.961849 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:36:42.961849 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:42.961849 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:36:42.961849 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:42.961849 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:36:43.386304 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:36:43.387087 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:36:43.387288 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:36:43.388532 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:36:43.388923 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:43.389944 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:36:43.389944 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:43.389944 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:36:43.389944 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:36:43.389944 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:43.389944 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:36:43.389944 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:43.389944 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:36:43.391659 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:43.391659 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:36:43.391659 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:43.391659 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:36:43.391659 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:36:43.391659 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:36:43.391659 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:36:43.391659 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:43.391659 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:43.391659 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:43.391659 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:36:43.391659 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:43.392070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:43.392070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:43.392070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:36:43.392070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:43.392070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:36:43.392070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:36:43.392070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:43.392070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:36:43.392070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:43.392070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:36:43.392070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:36:43.392070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:43.392070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:36:43.392479 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:43.392479 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:36:43.393811 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:43.393811 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:36:43.393811 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:36:43.393811 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:36:43.393811 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:36:43.393811 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:36:43.393811 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:43.393811 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:36:43.393811 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:36:43.393811 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:43.393811 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:36:43.393811 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:36:43.393811 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:43.393811 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:43.393811 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:36:43.393811 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:43.394307 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:36:43.394307 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:36:43.394307 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:43.394307 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:43.394307 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:43.394307 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:36:43.394307 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:43.394307 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:43.394307 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:43.394307 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:36:43.394307 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:43.394307 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:36:43.394307 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:36:43.394307 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:43.394307 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:36:43.394307 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:43.394307 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:36:43.394307 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:36:43.394307 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:43.394866 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:36:43.394866 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:43.394866 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:36:43.394866 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:36:43.394866 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:36:43.394866 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:36:43.394866 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:36:43.395183 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:36:43.395558 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:43.396403 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:36:43.396403 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:43.396403 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:36:43.396403 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:36:43.396403 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:43.396403 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:36:43.396403 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:43.396403 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:36:43.397078 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:43.397078 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:36:43.397078 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:36:43.397078 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:36:43.397078 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:43.397078 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:36:43.397078 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:43.397078 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:36:43.397078 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:43.397078 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:36:43.397078 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:43.397078 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:36:43.397078 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:43.397078 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:36:43.397078 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:36:43.397573 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:43.397573 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:36:43.397573 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:43.397573 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:36:43.397573 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:36:43.397573 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:43.397573 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:36:43.397573 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:43.397573 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:36:43.397937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:43.397937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:36:43.397937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:43.397937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:36:43.397937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:36:43.397937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:36:43.397937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:36:43.397937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:36:43.397937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:36:43.397937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:36:43.397937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:43.397937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:43.398300 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:43.398300 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:36:43.398300 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:43.398300 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:43.398300 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:43.398300 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:36:43.398300 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:43.398300 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:36:43.398300 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:36:43.398300 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:43.398300 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:36:43.398300 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:43.398300 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:36:43.398300 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:36:43.398721 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:43.398721 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:36:43.398721 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:43.398721 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:36:43.398973 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:43.398973 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:36:43.398973 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:36:43.398973 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:36:43.398973 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:36:43.398973 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:36:43.398973 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:43.398973 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:36:43.398973 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:36:43.398973 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:43.398973 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:36:43.398973 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:36:43.398973 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:43.398973 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:43.398973 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:36:43.398973 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:43.399503 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:36:43.399503 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:36:43.399503 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:36:43.399503 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:36:43.399503 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:36:43.399503 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:43.399503 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:43.399503 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:43.399503 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:36:43.399503 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:43.399503 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:43.399503 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:43.399503 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:36:43.399503 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:43.399503 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:36:43.399503 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:36:43.399503 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:43.399503 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:36:43.399998 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:43.399998 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:36:43.399998 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:36:43.399998 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:43.399998 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:36:43.399998 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:43.399998 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:36:43.402857 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:36:43.403168 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:36:43.403273 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:36:43.403437 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:36:43.403943 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:36:43.404342 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:43.405088 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:36:43.405088 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:43.405088 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:36:43.405088 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:36:43.405088 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:43.405088 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:36:43.405088 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:43.405088 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:36:43.405920 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:43.405920 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:36:43.405920 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:36:43.405920 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:36:43.405920 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:43.405920 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:36:43.405920 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:43.405920 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:36:43.405920 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:43.405920 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:36:43.405920 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:43.405920 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:36:43.405920 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:43.405920 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:36:43.405920 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:36:43.406379 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:43.406379 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:36:43.406379 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:43.406379 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:36:43.406379 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:36:43.406379 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:43.406379 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:36:43.406379 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:43.406379 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:36:43.406782 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:43.406782 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:36:43.406782 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:43.406782 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:36:43.406782 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:36:43.406782 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:36:43.406782 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:36:43.406782 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:36:43.406782 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:36:43.406782 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:43.406782 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:43.407113 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:43.407113 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:36:43.407113 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:43.407113 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:43.407113 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:43.407113 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:36:43.407113 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:43.407113 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:36:43.407113 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:36:43.407113 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:43.407113 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:36:43.407113 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:43.407113 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:36:43.407113 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:36:43.407512 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:43.407512 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:36:43.407512 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:43.407512 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:36:43.407846 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:43.407846 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:36:43.407846 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:36:43.407846 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:36:43.407846 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:36:43.407846 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:36:43.407846 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:43.407846 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:36:43.407846 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:36:43.407846 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:43.407846 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:36:43.407846 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:36:43.407846 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:43.407846 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:43.407846 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:36:43.407846 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:43.407846 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:36:43.408478 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:36:43.408478 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:36:43.408478 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:36:43.408478 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:43.408478 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:43.408478 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:43.408478 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:36:43.408478 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:43.408478 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:43.408478 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:43.408478 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:36:43.408478 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:43.408478 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:36:43.408478 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:36:43.408478 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:43.408478 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:36:43.408478 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:43.408478 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:36:43.408478 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:36:43.408957 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:43.408957 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:36:43.408957 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:43.408957 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:36:43.413112 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:36:43.413440 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:36:43.413603 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:36:43.413753 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:36:43.414228 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:36:43.414628 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:36:43.414826 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:43.415596 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:36:43.415596 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:43.415596 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:36:43.415596 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:36:43.415596 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:43.415596 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:36:43.415596 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:43.415596 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:36:43.416451 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:43.416451 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:36:43.416451 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:43.416451 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:36:43.416451 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:36:43.416451 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:36:43.416451 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:36:43.416451 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:43.416451 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:43.416451 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:43.416451 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:36:43.416451 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:43.416837 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:43.416837 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:43.416837 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:36:43.416837 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:43.416837 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:36:43.416837 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:36:43.416837 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:43.416837 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:36:43.416837 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:43.416837 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:36:43.416837 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:36:43.416837 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:43.416837 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:36:43.417187 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:43.417187 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:36:43.417418 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:43.417418 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:36:43.417418 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:36:43.417418 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:36:43.417418 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:36:43.417418 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:36:43.417418 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:43.417418 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:36:43.417418 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:36:43.417418 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:43.417418 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:36:43.417418 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:36:43.417418 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:43.417418 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:43.417418 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:36:43.417418 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:43.417917 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:36:43.417917 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:36:43.417917 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:43.417917 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:43.417917 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:43.417917 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:36:43.417917 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:43.417917 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:43.417917 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:43.417917 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:36:43.417917 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:43.417917 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:36:43.417917 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:36:43.417917 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:43.417917 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:36:43.417917 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:43.417917 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:36:43.417917 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:36:43.417917 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:43.418464 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:36:43.418464 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:43.418464 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:36:47.923395 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:36:47.924382 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:36:47.924669 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:36:47.926017 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:36:47.928865 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG dbcounter [-] [106123] Writing DB stats neutron:SELECT=240 {{(pid=106123) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} May 18 18:36:47.962183 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:36:47.962679 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:47.963677 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:36:47.963677 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:47.963677 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:36:47.963677 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:36:47.963677 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:47.963677 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:36:47.963677 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:47.963677 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:36:47.964613 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.039 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:47.964613 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:36:47.964613 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:47.964613 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:36:47.964613 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:36:47.964613 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:36:47.964613 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:36:47.964613 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:47.964613 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:47.964613 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:47.964613 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:36:47.964613 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:47.965145 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:47.965145 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:47.965145 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:36:47.965145 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:47.965145 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:36:47.965145 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:36:47.965145 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:47.965145 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:36:47.965145 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:47.965145 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:36:47.965145 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:36:47.965145 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:47.965145 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:36:47.965655 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:47.965655 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:36:47.965655 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:47.965655 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:36:47.965655 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:36:47.965655 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:36:47.965655 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:36:47.965655 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:36:47.965655 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:47.965655 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:36:47.965655 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:36:47.965655 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:47.965655 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:36:47.965655 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:36:47.965655 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:47.966304 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:47.966304 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:36:47.966304 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:47.966304 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:36:47.966304 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:36:47.966304 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:47.966304 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:47.966304 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:47.966304 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:36:47.966304 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:47.966304 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:47.966304 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:47.966304 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:36:47.966304 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:47.966304 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:36:47.966304 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:36:47.966304 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:47.966304 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:36:47.966304 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:47.967063 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:36:47.967063 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:36:47.967063 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:47.967063 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:36:47.967063 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:47.967063 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:36:48.387449 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:36:48.387939 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:36:48.388053 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:36:48.389232 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:36:48.389629 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:48.390684 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:36:48.390684 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:48.390684 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:36:48.390684 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:36:48.390684 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:48.390684 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:36:48.390684 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:48.390684 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:36:48.392350 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:48.392350 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:36:48.392350 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:48.392350 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:36:48.392350 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:36:48.392350 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:36:48.392350 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:36:48.392350 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:48.392350 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:48.392350 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:48.392350 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:36:48.392350 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:48.392926 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:48.392926 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:48.392926 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:36:48.392926 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:48.392926 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:36:48.392926 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:36:48.392926 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:48.392926 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:36:48.392926 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:48.392926 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:36:48.392926 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:36:48.392926 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:48.392926 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:36:48.393486 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:48.393486 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:36:48.394022 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:48.394022 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:36:48.394022 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:36:48.394022 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:36:48.394022 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:36:48.394022 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:36:48.394022 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:48.394022 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:36:48.394022 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:36:48.394022 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:48.394022 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:36:48.394022 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:36:48.394022 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:48.394022 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:48.394022 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:36:48.394022 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:48.394784 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:36:48.394784 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:36:48.394784 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:48.394784 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:48.394784 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:48.394784 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:36:48.394784 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:48.394784 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:48.394784 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:48.394784 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:36:48.394784 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:48.394784 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:36:48.394784 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:36:48.394784 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:48.394784 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:36:48.394784 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:48.394784 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:36:48.394784 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:36:48.394784 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:48.395711 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:36:48.395711 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:48.395711 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:36:48.395711 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:36:48.395711 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:36:48.395711 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:36:48.395711 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:36:48.396022 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:36:48.396022 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:48.396657 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:36:48.396657 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:48.396657 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:36:48.396657 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:36:48.396657 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:48.396657 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:36:48.396657 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:48.396657 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:36:48.397866 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:48.397866 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:36:48.397866 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:36:48.397866 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:36:48.397866 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:48.397866 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:36:48.397866 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:48.397866 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:36:48.397866 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:48.397866 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:36:48.397866 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:48.397866 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:36:48.397866 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:48.397866 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:36:48.397866 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:36:48.398625 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:48.398625 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:36:48.398625 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:48.398625 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:36:48.398625 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:36:48.398625 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:48.398625 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:36:48.398625 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:48.398625 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:36:48.399233 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:48.399233 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:36:48.399233 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:48.399233 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:36:48.399233 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:36:48.399233 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:36:48.399233 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:36:48.399233 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:36:48.399233 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:36:48.399233 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:36:48.399233 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:48.399233 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:48.400101 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:48.400101 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:36:48.400101 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:48.400101 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:48.400101 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:48.400101 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:36:48.400101 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:48.400101 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:36:48.400101 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:36:48.400101 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:48.400101 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:36:48.400101 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:48.400101 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:36:48.400101 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:36:48.400820 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:48.400820 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:36:48.400820 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:48.400820 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:36:48.400820 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:48.400820 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:36:48.400820 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:36:48.400820 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:36:48.400820 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:36:48.400820 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:36:48.400820 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:48.400820 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:36:48.400820 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:36:48.400820 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:48.401520 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:36:48.401520 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:36:48.401520 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:48.401520 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:48.401520 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:36:48.401520 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:48.401520 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:36:48.401520 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:36:48.401520 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:36:48.401520 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:36:48.401520 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:36:48.401520 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:48.401520 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:48.401520 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:48.401520 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:36:48.401520 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:48.401520 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:48.401520 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:48.401520 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:36:48.402673 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:48.402673 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:36:48.402673 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:36:48.402673 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:48.402673 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:36:48.402673 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:48.402673 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:36:48.402673 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:36:48.402673 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:48.402673 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:36:48.402673 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:48.402673 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:36:48.403414 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:36:48.403791 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:36:48.403903 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:36:48.404036 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:36:48.404388 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:36:48.404605 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:48.405103 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:36:48.405103 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:48.405103 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:36:48.405103 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:36:48.405103 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:48.405103 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:36:48.405103 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:48.405103 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:36:48.406014 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:48.406014 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:36:48.406014 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:36:48.406014 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:36:48.406014 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:48.406014 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:36:48.406014 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:48.406014 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:36:48.406014 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:48.406014 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:36:48.406014 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:48.406014 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:36:48.406014 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:48.406014 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:36:48.406014 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:36:48.406449 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:48.406449 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:36:48.406449 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:48.406449 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:36:48.406449 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:36:48.406449 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:48.406449 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:36:48.406449 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:48.406449 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:36:48.407353 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:48.407353 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:36:48.407353 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:48.407353 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:36:48.407353 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:36:48.407353 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:36:48.407353 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:36:48.407353 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:36:48.407353 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:36:48.407353 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:48.407353 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:48.408104 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:48.408104 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:36:48.408104 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:48.408104 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:48.408104 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:48.408104 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:36:48.408104 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:48.408104 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:36:48.408104 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:36:48.408104 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:48.408104 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:36:48.408104 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:48.408104 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:36:48.408104 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:36:48.408776 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:48.408776 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:36:48.408776 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:48.408776 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:36:48.408776 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:48.408776 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:36:48.408776 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:36:48.408776 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:36:48.408776 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:36:48.408776 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:36:48.408776 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:48.408776 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:36:48.408776 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:36:48.408776 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:48.409485 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:36:48.409485 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:36:48.409485 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:48.409485 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:48.409485 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:36:48.409485 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:48.409485 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:36:48.409485 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:36:48.409485 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:36:48.409485 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:36:48.409485 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:48.409485 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:48.409485 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:48.409485 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:36:48.409485 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:48.409485 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:48.409485 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:48.409485 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:36:48.409485 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:48.410362 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:36:48.410362 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:36:48.410362 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:48.410362 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:36:48.410362 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:48.410362 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:36:48.410362 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:36:48.410362 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:48.410362 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:36:48.410362 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:48.410362 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:36:48.413681 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:36:48.413993 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:36:48.414095 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:36:48.414267 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:36:48.414716 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:36:48.414986 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:36:48.415080 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:48.415768 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:36:48.415768 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:48.415768 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:36:48.415768 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:36:48.415768 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:48.415768 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:36:48.415768 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:48.415768 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:36:48.417465 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:48.417465 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:36:48.417465 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:48.417465 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:36:48.417465 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:36:48.417465 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:36:48.417465 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:36:48.417465 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:48.417465 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:48.417465 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:48.417465 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:36:48.417465 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:48.417825 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:48.417825 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:48.417825 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:36:48.417825 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:48.417825 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:36:48.417825 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:36:48.417825 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:48.417825 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:36:48.417825 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:48.417825 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:36:48.417825 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:36:48.417825 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:48.417825 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:36:48.418158 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:48.418158 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:36:48.419953 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:48.419953 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:36:48.419953 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:36:48.419953 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:36:48.419953 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:36:48.419953 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:36:48.419953 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:48.419953 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:36:48.419953 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:36:48.419953 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:48.419953 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:36:48.419953 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:36:48.419953 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:48.419953 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:48.419953 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:36:48.419953 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:48.420427 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:36:48.420427 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:36:48.420427 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:48.420427 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:48.420427 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:48.420427 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:36:48.420427 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:48.420427 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:48.420427 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:48.420427 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:36:48.420427 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:48.420427 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:36:48.420427 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:36:48.420427 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:48.420427 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:36:48.420427 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:48.420427 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:36:48.420427 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:36:48.420427 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:48.420894 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:36:48.420894 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:48.420894 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:36:52.924268 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:36:52.925048 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:36:52.925048 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:36:52.926565 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:36:52.963073 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:36:52.963509 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:52.964545 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:36:52.964545 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:52.964545 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:36:52.964545 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:36:52.964545 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:52.964545 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:36:52.964545 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:52.964545 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:36:52.965462 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.040 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:52.965462 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:36:52.965462 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:52.965462 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:36:52.965462 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:36:52.965462 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:36:52.965462 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:36:52.965462 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:52.965462 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:52.965462 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:52.965462 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:36:52.965462 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:52.965993 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:52.965993 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:52.965993 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:36:52.965993 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:52.965993 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:36:52.965993 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:36:52.965993 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:52.965993 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:36:52.965993 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:52.965993 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:36:52.965993 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:36:52.965993 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:52.965993 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:36:52.966638 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:52.966638 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:36:52.966638 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:52.966638 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:36:52.966638 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:36:52.966638 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:36:52.966638 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:36:52.966638 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:36:52.966638 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:52.966638 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:36:52.966638 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:36:52.966638 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:52.966638 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:36:52.966638 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:36:52.966638 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:52.967257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:52.967257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:36:52.967257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:52.967257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:36:52.967257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:36:52.967257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:52.967257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:52.967257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:52.967257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:36:52.967257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:52.967257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:52.967257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:52.967257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:36:52.967257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:52.967257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:36:52.967257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:36:52.967257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:52.967257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:36:52.967257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:52.968265 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:36:52.968265 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:36:52.968265 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:52.968265 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:36:52.968265 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:52.968265 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:36:53.388173 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:36:53.388750 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:36:53.388946 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:36:53.390047 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:36:53.390295 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:53.391069 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:36:53.391069 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:53.391069 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:36:53.391069 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:36:53.391069 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:53.391069 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:36:53.391069 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:53.391069 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:36:53.392701 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:53.392701 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:36:53.392701 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:53.392701 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:36:53.392701 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:36:53.392701 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:36:53.392701 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:36:53.392701 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:53.392701 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:53.392701 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:53.392701 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:36:53.392701 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:53.393236 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:53.393236 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:53.393236 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:36:53.393236 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:53.393236 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:36:53.393236 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:36:53.393236 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:53.393236 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:36:53.393236 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:53.393236 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:36:53.393236 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:36:53.393236 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:53.393236 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:36:53.393824 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:53.393824 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:36:53.394413 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:53.394413 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:36:53.394413 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:36:53.394413 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:36:53.394413 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:36:53.394413 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:36:53.394413 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:53.394413 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:36:53.394413 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:36:53.394413 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:53.394413 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:36:53.394413 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:36:53.394413 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:53.394413 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:53.394413 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:36:53.394413 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:53.395168 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:36:53.395168 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:36:53.395168 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:53.395168 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:53.395168 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:53.395168 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:36:53.395168 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:53.395168 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:53.395168 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:53.395168 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:36:53.395168 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:53.395168 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:36:53.395168 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:36:53.395168 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:53.395168 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:36:53.395168 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:53.395168 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:36:53.395168 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:36:53.395168 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:53.395775 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:36:53.395775 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:53.395775 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:36:53.395775 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:36:53.395775 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:36:53.395775 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:36:53.395775 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:36:53.396066 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:36:53.396300 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:53.396849 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:36:53.396849 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:53.396849 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:36:53.396849 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:36:53.396849 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:53.396849 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:36:53.396849 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:53.396849 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:36:53.397709 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:53.397709 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:36:53.397709 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:36:53.397709 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:36:53.397709 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:53.397709 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:36:53.397709 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:53.397709 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:36:53.397709 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:53.397709 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:36:53.397709 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:53.397709 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:36:53.397709 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:53.397709 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:36:53.397709 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:36:53.398177 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:53.398177 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:36:53.398177 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:53.398177 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:36:53.398177 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:36:53.398177 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:53.398177 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:36:53.398177 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:53.398177 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:36:53.398467 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:53.398467 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:36:53.398467 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:53.398467 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:36:53.398467 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:36:53.398467 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:36:53.398467 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:36:53.398467 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:36:53.398467 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:36:53.398467 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:36:53.398467 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:53.398467 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:53.398805 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:53.398805 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:36:53.398805 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:53.398805 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:53.398805 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:53.398805 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:36:53.398805 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:53.398805 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:36:53.398805 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:36:53.398805 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:53.398805 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:36:53.398805 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:53.398805 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:36:53.398805 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:36:53.399211 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:53.399211 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:36:53.399211 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:53.399211 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:36:53.399508 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:53.399508 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:36:53.399508 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:36:53.399508 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:36:53.399508 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:36:53.399508 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:36:53.399508 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:53.399508 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:36:53.399508 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:36:53.399508 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:53.399508 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:36:53.399508 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:36:53.399508 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:53.399508 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:53.399508 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:36:53.399508 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:53.399924 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:36:53.399924 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:36:53.399924 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:36:53.399924 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:36:53.399924 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:36:53.399924 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:53.399924 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:53.399924 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:53.399924 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:36:53.399924 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:53.399924 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:53.399924 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:53.399924 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:36:53.399924 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:53.399924 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:36:53.399924 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:36:53.399924 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:53.399924 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:36:53.400465 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:53.400465 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:36:53.400465 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:36:53.400465 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:53.400465 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:36:53.400465 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:53.400465 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:36:53.403878 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:36:53.404119 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:36:53.404229 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:36:53.404376 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:36:53.404912 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:36:53.405300 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:53.406073 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:36:53.406073 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:53.406073 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:36:53.406073 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:36:53.406073 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:53.406073 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:36:53.406073 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:53.406073 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:36:53.406827 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:53.406827 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:36:53.406827 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:36:53.406827 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:36:53.406827 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:53.406827 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:36:53.406827 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:53.406827 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:36:53.406827 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:53.406827 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:36:53.406827 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:53.406827 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:36:53.406827 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:53.406827 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:36:53.406827 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:36:53.407252 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:53.407252 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:36:53.407252 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:53.407252 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:36:53.407252 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:36:53.407252 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:53.407252 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:36:53.407252 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:53.407252 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:36:53.407774 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:53.407774 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:36:53.407774 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:53.407774 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:36:53.407774 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:36:53.407774 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:36:53.407774 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:36:53.407774 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:36:53.407774 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:36:53.407774 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:53.407774 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:53.408071 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:53.408071 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:36:53.408071 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:53.408071 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:53.408071 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:53.408071 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:36:53.408071 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:53.408071 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:36:53.408071 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:36:53.408071 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:53.408071 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:36:53.408071 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:53.408071 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:36:53.408071 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:36:53.408455 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:53.408455 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:36:53.408455 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:53.408455 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:36:53.408818 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:53.408818 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:36:53.408818 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:36:53.408818 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:36:53.408818 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:36:53.408818 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:36:53.408818 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:53.408818 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:36:53.408818 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:36:53.408818 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:53.408818 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:36:53.408818 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:36:53.408818 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:53.408818 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:53.408818 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:36:53.408818 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:53.408818 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:36:53.409262 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:36:53.409262 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:36:53.409262 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:36:53.409262 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:53.409262 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:53.409262 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:53.409262 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:36:53.409262 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:53.409262 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:53.409262 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:53.409262 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:36:53.409262 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:53.409262 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:36:53.409262 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:36:53.409262 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:53.409262 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:36:53.409262 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:53.409262 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:36:53.409262 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:36:53.409741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:53.409741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:36:53.409741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:53.409741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:36:53.414110 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:36:53.414340 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:36:53.414481 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:36:53.414633 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:36:53.415191 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:36:53.415602 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:36:53.415796 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:53.416536 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:36:53.416536 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:53.416536 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:36:53.416536 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:36:53.416536 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:53.416536 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:36:53.416536 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:53.416536 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:36:53.417340 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:53.417340 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:36:53.417340 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:53.417340 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:36:53.417340 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:36:53.417340 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:36:53.417340 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:36:53.417340 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:53.417340 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:53.417340 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:53.417340 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:36:53.417340 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:53.417707 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:53.417707 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:53.417707 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:36:53.417707 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:53.417707 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:36:53.417707 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:36:53.417707 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:53.417707 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:36:53.417707 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:53.417707 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:36:53.417707 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:36:53.417707 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:53.417707 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:36:53.418061 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:53.418061 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:36:53.418279 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:53.418279 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:36:53.418279 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:36:53.418279 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:36:53.418279 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:36:53.418279 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:36:53.418279 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:53.418279 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:36:53.418279 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:36:53.418279 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:53.418279 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:36:53.418279 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:36:53.418279 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:53.418279 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:53.418279 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:36:53.418279 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:53.418809 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:36:53.418809 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:36:53.418809 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:53.418809 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:53.418809 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:53.418809 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:36:53.418809 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:53.418809 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:53.418809 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:53.418809 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:36:53.418809 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:53.418809 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:36:53.418809 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:36:53.418809 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:53.418809 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:36:53.418809 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:53.418809 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:36:53.418809 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:36:53.418809 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:53.419415 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:36:53.419415 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:53.419415 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:36:57.925055 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:36:57.925928 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:36:57.926099 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:36:57.927486 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:36:57.958145 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:36:57.958603 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:57.959798 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:36:57.959798 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:57.959798 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:36:57.959798 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:36:57.959798 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:57.959798 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:36:57.959798 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:57.959798 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:36:57.960696 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.034 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:57.960696 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:36:57.960696 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:57.960696 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:36:57.960696 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:36:57.960696 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:36:57.960696 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:36:57.960696 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:57.960696 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:57.960696 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:57.960696 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:36:57.960696 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:57.961487 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:57.961487 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:57.961487 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:36:57.961487 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:57.961487 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:36:57.961487 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:36:57.961487 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:57.961487 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:36:57.961487 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:57.961487 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:36:57.961487 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:36:57.961487 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:57.961487 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:36:57.962257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:57.962257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:36:57.962257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:57.962257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:36:57.962257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:36:57.962257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:36:57.962257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:36:57.962257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:36:57.962257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:57.962257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:36:57.962257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:36:57.962257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:57.962257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:36:57.962257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:36:57.962257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:57.963179 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:57.963179 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:36:57.963179 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:57.963179 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:36:57.963179 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:36:57.963179 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:57.963179 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:57.963179 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:57.963179 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:36:57.963179 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:57.963179 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:57.963179 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:57.963179 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:36:57.963179 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:57.963179 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:36:57.963179 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:36:57.963179 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:57.963179 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:36:57.963179 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:57.964433 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:36:57.964433 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:36:57.964433 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:57.964433 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:36:57.964433 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:57.964433 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:36:58.388914 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:36:58.389556 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:36:58.389796 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:36:58.390861 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:36:58.391161 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:58.392123 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:36:58.392123 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:58.392123 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:36:58.392123 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:36:58.392123 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:58.392123 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:36:58.392123 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:58.392123 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:36:58.393847 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:58.393847 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:36:58.393847 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:58.393847 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:36:58.393847 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:36:58.393847 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:36:58.393847 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:36:58.393847 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:58.393847 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:58.393847 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:58.393847 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:36:58.393847 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:58.394399 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:58.394399 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:58.394399 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:36:58.394399 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:58.394399 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:36:58.394399 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:36:58.394399 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:58.394399 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:36:58.394399 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:58.394399 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:36:58.394399 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:36:58.394399 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:58.394399 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:36:58.394923 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:58.394923 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:36:58.395923 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:58.395923 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:36:58.395923 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:36:58.395923 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:36:58.395923 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:36:58.395923 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:36:58.395923 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:58.395923 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:36:58.395923 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:36:58.395923 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:58.395923 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:36:58.395923 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:36:58.395923 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:58.395923 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:58.395923 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:36:58.395923 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:58.396608 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:36:58.396608 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:36:58.396608 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:58.396608 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:58.396608 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:58.396608 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:36:58.396608 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:58.396608 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:58.396608 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:58.396608 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:36:58.396608 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:58.396608 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:36:58.396608 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:36:58.396608 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:58.396608 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:36:58.396608 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:58.396608 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:36:58.396608 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:36:58.396608 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:58.397392 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:36:58.397392 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:58.397392 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:36:58.397392 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:36:58.397392 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:36:58.397392 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:36:58.397392 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:36:58.397732 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:36:58.398020 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:58.398731 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:36:58.398731 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:58.398731 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:36:58.398731 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:36:58.398731 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:58.398731 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:36:58.398731 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:58.398731 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:36:58.399519 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:58.399519 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:36:58.399519 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:36:58.399519 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:36:58.399519 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:58.399519 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:36:58.399519 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:58.399519 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:36:58.399519 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:58.399519 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:36:58.399519 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:58.399519 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:36:58.399519 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:58.399519 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:36:58.399519 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:36:58.399987 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:58.399987 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:36:58.399987 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:58.399987 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:36:58.399987 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:36:58.399987 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:58.399987 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:36:58.399987 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:58.399987 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:36:58.400569 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:58.400569 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:36:58.400569 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:58.400569 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:36:58.400569 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:36:58.400569 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:36:58.400569 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:36:58.400569 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:36:58.400569 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:36:58.400569 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:36:58.400569 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:58.400569 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:58.400944 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:58.400944 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:36:58.400944 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:58.400944 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:58.400944 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:58.400944 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:36:58.400944 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:58.400944 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:36:58.400944 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:36:58.400944 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:58.400944 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:36:58.400944 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:58.400944 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:36:58.400944 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:36:58.401349 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:58.401349 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:36:58.401349 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:58.401349 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:36:58.401823 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:58.401823 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:36:58.401823 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:36:58.401823 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:36:58.401823 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:36:58.401823 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:36:58.401823 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:58.401823 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:36:58.401823 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:36:58.401823 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:58.401823 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:36:58.401823 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:36:58.401823 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:58.401823 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:58.401823 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:36:58.401823 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:58.402279 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:36:58.402279 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:36:58.402279 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:36:58.402279 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:36:58.402279 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:36:58.402279 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:58.402279 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:58.402279 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:58.402279 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:36:58.402279 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:58.402279 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:58.402279 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:58.402279 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:36:58.402279 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:58.402279 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:36:58.402279 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:36:58.402279 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:58.402279 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:36:58.402860 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:58.402860 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:36:58.402860 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:36:58.402860 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:58.402860 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:36:58.402860 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:58.402860 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:36:58.404225 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:36:58.404500 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:36:58.404635 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:36:58.404794 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:36:58.405221 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:36:58.405683 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:58.406333 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:36:58.406333 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:58.406333 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:36:58.406333 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:36:58.406333 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:58.406333 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:36:58.406333 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:58.406333 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:36:58.407106 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:58.407106 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:36:58.407106 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:36:58.407106 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:36:58.407106 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:58.407106 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:36:58.407106 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:58.407106 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:36:58.407106 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:58.407106 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:36:58.407106 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:58.407106 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:36:58.407106 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:58.407106 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:36:58.407106 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:36:58.407561 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:58.407561 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:36:58.407561 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:58.407561 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:36:58.407561 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:36:58.407561 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:58.407561 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:36:58.407561 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:58.407561 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:36:58.408184 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:58.408184 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:36:58.408184 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:58.408184 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:36:58.408184 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:36:58.408184 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:36:58.408184 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:36:58.408184 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:36:58.408184 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:36:58.408184 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:58.408184 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:58.408562 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:58.408562 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:36:58.408562 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:58.408562 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:58.408562 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:58.408562 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:36:58.408562 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:58.408562 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:36:58.408562 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:36:58.408562 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:58.408562 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:36:58.408562 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:58.408562 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:36:58.408562 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:36:58.408944 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:58.408944 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:36:58.408944 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:58.408944 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:36:58.409507 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:58.409507 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:36:58.409507 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:36:58.409507 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:36:58.409507 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:36:58.409507 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:36:58.409507 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:58.409507 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:36:58.409507 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:36:58.409507 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:58.409507 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:36:58.409507 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:36:58.409507 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:58.409507 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:58.409507 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:36:58.409507 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:58.409507 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:36:58.410006 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:36:58.410006 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:36:58.410006 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:36:58.410006 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:58.410006 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:58.410006 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:58.410006 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:36:58.410006 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:58.410006 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:58.410006 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:58.410006 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:36:58.410006 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:58.410006 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:36:58.410006 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:36:58.410006 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:58.410006 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:36:58.410006 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:58.410006 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:36:58.410006 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:36:58.410560 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:58.410560 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:36:58.410560 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:58.410560 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:36:58.414393 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:36:58.414635 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:36:58.414774 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:36:58.414960 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:36:58.415236 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:36:58.415499 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:36:58.415702 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:58.416333 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:36:58.416333 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:58.416333 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:36:58.416333 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:36:58.416333 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:58.416333 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:36:58.416333 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:58.416333 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:36:58.417199 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:58.417199 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:36:58.417199 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:58.417199 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:36:58.417199 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:36:58.417199 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:36:58.417199 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:36:58.417199 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:58.417199 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:58.417199 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:58.417199 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:36:58.417199 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:58.417553 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:36:58.417553 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:58.417553 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:36:58.417553 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:58.417553 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:36:58.417553 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:36:58.417553 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:58.417553 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:36:58.417553 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:58.417553 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:36:58.417553 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:36:58.417553 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:58.417553 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:36:58.417880 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:58.417880 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:36:58.418386 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:58.418386 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:36:58.418386 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:36:58.418386 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:36:58.418386 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:36:58.418386 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:36:58.418386 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:58.418386 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:36:58.418386 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:36:58.418386 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:58.418386 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:36:58.418386 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:36:58.418386 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:36:58.418386 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:36:58.418386 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:36:58.418386 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:36:58.418861 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:36:58.418861 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:36:58.418861 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:58.418861 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:58.418861 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:36:58.418861 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:36:58.418861 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:36:58.418861 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:36:58.418861 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:36:58.418861 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:36:58.418861 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:36:58.418861 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:36:58.418861 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:36:58.418861 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:36:58.418861 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:36:58.418861 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:36:58.418861 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:36:58.418861 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:36:58.418861 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:36:58.419412 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:36:58.419412 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:36:58.419412 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:37:02.925839 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:37:02.926725 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:37:02.927043 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:37:02.928729 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:37:02.962506 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:37:02.962964 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:02.964073 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:37:02.964073 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:02.964073 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:37:02.964073 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:37:02.964073 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:02.964073 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:37:02.964073 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:02.964073 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:37:02.965057 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.037 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:02.965057 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:37:02.965057 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:02.965057 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:37:02.965057 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:37:02.965057 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:37:02.965057 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:37:02.965057 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:02.965057 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:02.965057 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:02.965057 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:37:02.965057 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:02.965956 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:02.965956 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:02.965956 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:37:02.965956 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:02.965956 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:37:02.965956 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:37:02.965956 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:02.965956 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:37:02.965956 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:02.965956 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:37:02.965956 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:37:02.965956 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:02.965956 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:37:02.966717 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:02.966717 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:37:02.966717 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:02.966717 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:37:02.966717 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:37:02.966717 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:37:02.966717 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:37:02.966717 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:37:02.966717 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:02.966717 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:37:02.966717 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:37:02.966717 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:02.966717 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:37:02.966717 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:37:02.966717 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:02.967401 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:02.967401 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:37:02.967401 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:02.967401 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:37:02.967401 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:37:02.967401 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:02.967401 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:02.967401 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:02.967401 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:37:02.967401 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:02.967401 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:02.967401 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:02.967401 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:37:02.967401 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:02.967401 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:37:02.967401 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:37:02.967401 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:02.967401 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:37:02.967401 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:02.968232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:37:02.968232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:37:02.968232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:02.968232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:37:02.968232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:02.968232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:37:03.389731 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:37:03.390517 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:37:03.390717 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:37:03.392076 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:37:03.392589 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:03.393687 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:37:03.393687 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:03.393687 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:37:03.393687 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:37:03.393687 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:03.393687 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:37:03.393687 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:03.393687 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:37:03.395342 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:03.395342 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:37:03.395342 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:03.395342 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:37:03.395342 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:37:03.395342 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:37:03.395342 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:37:03.395342 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:03.395342 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:03.395342 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:03.395342 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:37:03.395342 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:03.395935 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:03.395935 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:03.395935 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:37:03.395935 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:03.395935 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:37:03.395935 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:37:03.395935 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:03.395935 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:37:03.395935 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:03.395935 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:37:03.395935 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:37:03.395935 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:03.395935 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:37:03.396566 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:03.396566 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:37:03.397038 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:03.397038 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:37:03.397038 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:37:03.397038 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:37:03.397038 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:37:03.397038 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:37:03.397038 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:03.397038 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:37:03.397038 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:37:03.397038 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:03.397038 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:37:03.397038 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:37:03.397038 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:03.397038 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:03.397038 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:37:03.397038 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:03.397788 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:37:03.397788 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:37:03.397788 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:03.397788 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:03.397788 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:03.397788 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:37:03.397788 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:03.397788 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:03.397788 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:03.397788 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:37:03.397788 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:03.397788 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:37:03.397788 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:37:03.397788 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:03.397788 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:37:03.397788 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:03.397788 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:37:03.397788 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:37:03.397788 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:03.398798 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:37:03.398798 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:03.398798 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:37:03.398798 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:37:03.398798 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:37:03.398798 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:37:03.398798 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:37:03.398798 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:37:03.399362 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:03.399776 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:37:03.399776 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:03.399776 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:37:03.399776 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:37:03.399776 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:03.399776 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:37:03.399776 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:03.399776 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:37:03.401025 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:03.401025 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:37:03.401025 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:37:03.401025 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:37:03.401025 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:03.401025 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:37:03.401025 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:03.401025 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:37:03.401025 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:03.401025 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:37:03.401025 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:03.401025 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:37:03.401025 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:03.401025 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:37:03.401025 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:37:03.401879 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:03.401879 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:37:03.401879 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:03.401879 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:37:03.401879 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:37:03.401879 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:03.401879 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:37:03.401879 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:03.401879 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:37:03.402675 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:03.402675 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:37:03.402675 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:03.402675 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:37:03.402675 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:37:03.402675 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:37:03.402675 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:37:03.402675 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:37:03.402675 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:37:03.402675 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:37:03.402675 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:03.402675 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:03.403426 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:03.403426 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:37:03.403426 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:03.403426 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:03.403426 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:03.403426 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:37:03.403426 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:03.403426 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:37:03.403426 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:37:03.403426 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:03.403426 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:37:03.403426 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:03.403426 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:37:03.403426 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:37:03.404000 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:03.404000 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:37:03.404000 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:03.404000 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:37:03.404138 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:03.404138 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:37:03.404138 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:37:03.404138 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:37:03.404138 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:37:03.404138 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:37:03.404138 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:03.404138 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:37:03.404138 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:37:03.404138 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:03.404138 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:37:03.404138 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:37:03.404138 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:03.404138 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:03.404138 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:37:03.404138 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:03.404600 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:37:03.404600 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:37:03.404600 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:37:03.404600 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:37:03.404600 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:37:03.404600 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:03.404600 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:03.404600 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:03.404600 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:37:03.404600 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:03.404600 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:03.404600 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:03.404600 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:37:03.404600 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:03.404600 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:37:03.404600 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:37:03.404600 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:03.404600 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:37:03.405125 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:03.405125 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:37:03.405125 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:37:03.405125 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:03.405125 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:37:03.405125 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:03.405125 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:37:03.405125 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:37:03.405125 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:37:03.405125 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:37:03.405462 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:37:03.405723 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:37:03.406082 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:03.406846 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:37:03.406846 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:03.406846 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:37:03.406846 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:37:03.406846 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:03.406846 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:37:03.406846 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:03.406846 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:37:03.407683 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:03.407683 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:37:03.407683 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:37:03.407683 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:37:03.407683 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:03.407683 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:37:03.407683 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:03.407683 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:37:03.407683 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:03.407683 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:37:03.407683 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:03.407683 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:37:03.407683 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:03.407683 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:37:03.407683 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:37:03.408210 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:03.408210 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:37:03.408210 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:03.408210 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:37:03.408210 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:37:03.408210 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:03.408210 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:37:03.408210 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:03.408210 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:37:03.408570 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:03.408570 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:37:03.408570 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:03.408570 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:37:03.408570 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:37:03.408570 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:37:03.408570 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:37:03.408570 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:37:03.408570 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:37:03.408570 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:03.408570 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:03.408930 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:03.408930 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:37:03.408930 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:03.408930 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:03.408930 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:03.408930 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:37:03.408930 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:03.408930 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:37:03.408930 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:37:03.408930 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:03.408930 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:37:03.408930 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:03.408930 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:37:03.408930 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:37:03.409334 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:03.409334 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:37:03.409334 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:03.409334 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:37:03.409932 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:03.409932 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:37:03.409932 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:37:03.409932 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:37:03.409932 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:37:03.409932 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:37:03.409932 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:03.409932 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:37:03.409932 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:37:03.409932 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:03.409932 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:37:03.409932 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:37:03.409932 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:03.409932 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:03.409932 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:37:03.409932 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:03.409932 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:37:03.410527 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:37:03.410527 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:37:03.410527 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:37:03.410527 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:03.410527 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:03.410527 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:03.410527 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:37:03.410527 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:03.410527 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:03.410527 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:03.410527 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:37:03.410527 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:03.410527 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:37:03.410527 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:37:03.410527 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:03.410527 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:37:03.410527 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:03.410527 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:37:03.410527 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:37:03.411084 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:03.411084 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:37:03.411084 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:03.411084 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:37:03.414799 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:37:03.415114 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:37:03.415229 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:37:03.415411 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:37:03.415892 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:37:03.416235 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:37:03.416480 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:03.417181 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:37:03.417181 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:03.417181 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:37:03.417181 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:37:03.417181 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:03.417181 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:37:03.417181 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:03.417181 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:37:03.418070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:03.418070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:37:03.418070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:03.418070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:37:03.418070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:37:03.418070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:37:03.418070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:37:03.418070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:03.418070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:03.418070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:03.418070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:37:03.418070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:03.418460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:03.418460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:03.418460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:37:03.418460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:03.418460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:37:03.418460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:37:03.418460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:03.418460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:37:03.418460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:03.418460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:37:03.418460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:37:03.418460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:03.418460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:37:03.418834 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:03.418834 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:37:03.419061 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:03.419061 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:37:03.419061 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:37:03.419061 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:37:03.419061 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:37:03.419061 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:37:03.419061 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:03.419061 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:37:03.419061 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:37:03.419061 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:03.419061 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:37:03.419061 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:37:03.419061 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:03.419061 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:03.419061 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:37:03.419061 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:03.419528 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:37:03.419528 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:37:03.419528 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:03.419528 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:03.419528 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:03.419528 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:37:03.419528 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:03.419528 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:03.419528 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:03.419528 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:37:03.419528 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:03.419528 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:37:03.419528 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:37:03.419528 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:03.419528 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:37:03.419528 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:03.419528 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:37:03.419528 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:37:03.419528 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:03.420163 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:37:03.420163 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:03.420163 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:37:07.926624 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:37:07.927503 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:37:07.927643 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:37:07.929062 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:37:07.965468 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:37:07.965468 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:07.966715 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:37:07.966715 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:07.966715 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:37:07.966715 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:37:07.966715 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:07.966715 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:37:07.966715 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:07.966715 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:37:07.967681 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.039 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:07.967681 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:37:07.967681 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:07.967681 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:37:07.967681 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:37:07.967681 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:37:07.967681 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:37:07.967681 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:07.967681 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:07.967681 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:07.967681 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:37:07.967681 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:07.968373 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:07.968373 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:07.968373 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:37:07.968373 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:07.968373 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:37:07.968373 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:37:07.968373 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:07.968373 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:37:07.968373 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:07.968373 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:37:07.968373 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:37:07.968373 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:07.968373 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:37:07.969159 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:07.969159 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:37:07.969159 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:07.969159 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:37:07.969159 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:37:07.969159 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:37:07.969159 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:37:07.969159 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:37:07.969159 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:07.969159 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:37:07.969159 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:37:07.969159 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:07.969159 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:37:07.969159 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:37:07.969159 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:07.970412 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:07.970412 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:37:07.970412 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:07.970412 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:37:07.970412 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:37:07.970412 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:07.970412 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:07.970412 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:07.970412 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:37:07.970412 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:07.970412 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:07.970412 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:07.970412 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:37:07.970412 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:07.970412 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:37:07.970412 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:37:07.970412 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:07.970412 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:37:07.970412 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:07.971907 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:37:07.971907 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:37:07.971907 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:07.971907 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:37:07.971907 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:07.971907 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:37:08.390526 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:37:08.391343 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:37:08.391595 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:37:08.392884 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:37:08.393356 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:08.394350 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:37:08.394350 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:08.394350 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:37:08.394350 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:37:08.394350 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:08.394350 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:37:08.394350 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:08.394350 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:37:08.395988 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:08.395988 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:37:08.395988 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:08.395988 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:37:08.395988 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:37:08.395988 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:37:08.395988 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:37:08.395988 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:08.395988 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:08.395988 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:08.395988 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:37:08.395988 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:08.396561 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:08.396561 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:08.396561 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:37:08.396561 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:08.396561 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:37:08.396561 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:37:08.396561 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:08.396561 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:37:08.396561 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:08.396561 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:37:08.396561 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:37:08.396561 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:08.396561 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:37:08.397136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:08.397136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:37:08.397785 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:08.397785 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:37:08.397785 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:37:08.397785 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:37:08.397785 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:37:08.397785 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:37:08.397785 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:08.397785 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:37:08.397785 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:37:08.397785 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:08.397785 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:37:08.397785 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:37:08.397785 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:08.397785 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:08.397785 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:37:08.397785 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:08.398559 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:37:08.398559 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:37:08.398559 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:08.398559 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:08.398559 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:08.398559 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:37:08.398559 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:08.398559 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:08.398559 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:08.398559 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:37:08.398559 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:08.398559 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:37:08.398559 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:37:08.398559 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:08.398559 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:37:08.398559 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:08.398559 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:37:08.398559 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:37:08.398559 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:08.399383 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:37:08.399383 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:08.399383 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:37:08.399383 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:37:08.399383 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:37:08.399383 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:37:08.399383 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:37:08.399747 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:37:08.399992 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:08.400755 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:37:08.400755 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:08.400755 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:37:08.400755 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:37:08.400755 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:08.400755 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:37:08.400755 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:08.400755 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:37:08.401552 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:08.401552 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:37:08.401552 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:37:08.401552 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:37:08.401552 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:08.401552 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:37:08.401552 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:08.401552 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:37:08.401552 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:08.401552 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:37:08.401552 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:08.401552 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:37:08.401552 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:08.401552 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:37:08.401552 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:37:08.401980 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:08.401980 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:37:08.401980 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:08.401980 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:37:08.401980 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:37:08.401980 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:08.401980 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:37:08.401980 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:08.401980 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:37:08.402451 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:08.402451 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:37:08.402451 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:08.402451 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:37:08.402451 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:37:08.402451 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:37:08.402451 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:37:08.402451 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:37:08.402451 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:37:08.402451 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:37:08.402451 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:08.402451 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:08.402808 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:08.402808 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:37:08.402808 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:08.402808 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:08.402808 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:08.402808 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:37:08.402808 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:08.402808 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:37:08.402808 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:37:08.402808 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:08.402808 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:37:08.402808 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:08.402808 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:37:08.402808 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:37:08.403196 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:08.403196 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:37:08.403196 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:08.403196 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:37:08.403457 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:08.403457 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:37:08.403457 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:37:08.403457 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:37:08.403457 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:37:08.403457 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:37:08.403457 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:08.403457 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:37:08.403457 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:37:08.403457 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:08.403457 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:37:08.403457 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:37:08.403457 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:08.403457 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:08.403457 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:37:08.403457 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:08.403919 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:37:08.403919 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:37:08.403919 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:37:08.403919 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:37:08.403919 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:37:08.403919 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:08.403919 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:08.403919 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:08.403919 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:37:08.403919 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:08.403919 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:08.403919 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:08.403919 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:37:08.403919 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:08.403919 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:37:08.403919 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:37:08.403919 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:08.403919 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:37:08.404445 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:08.404445 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:37:08.404445 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:37:08.404445 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:08.404445 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:37:08.404445 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:08.404445 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:37:08.405196 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:37:08.405561 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:37:08.405671 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:37:08.405800 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:37:08.406302 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:37:08.406824 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:08.407684 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:37:08.407684 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:08.407684 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:37:08.407684 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:37:08.407684 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:08.407684 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:37:08.407684 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:08.407684 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:37:08.408442 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:08.408442 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:37:08.408442 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:37:08.408442 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:37:08.408442 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:08.408442 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:37:08.408442 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:08.408442 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:37:08.408442 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:08.408442 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:37:08.408442 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:08.408442 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:37:08.408442 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:08.408442 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:37:08.408442 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:37:08.408882 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:08.408882 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:37:08.408882 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:08.408882 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:37:08.408882 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:37:08.408882 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:08.408882 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:37:08.408882 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:08.408882 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:37:08.409299 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:08.409299 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:37:08.409299 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:08.409299 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:37:08.409299 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:37:08.409299 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:37:08.409299 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:37:08.409299 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:37:08.409299 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:37:08.409299 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:08.409299 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:08.409648 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:08.409648 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:37:08.409648 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:08.409648 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:08.409648 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:08.409648 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:37:08.409648 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:08.409648 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:37:08.409648 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:37:08.409648 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:08.409648 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:37:08.409648 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:08.409648 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:37:08.409648 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:37:08.410033 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:08.410033 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:37:08.410033 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:08.410033 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:37:08.410370 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:08.410370 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:37:08.410370 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:37:08.410370 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:37:08.410370 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:37:08.410370 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:37:08.410370 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:08.410370 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:37:08.410370 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:37:08.410370 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:08.410370 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:37:08.410370 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:37:08.410370 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:08.410370 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:08.410370 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:37:08.410370 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:08.410370 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:37:08.410869 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:37:08.410869 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:37:08.410869 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:37:08.410869 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:08.410869 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:08.410869 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:08.410869 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:37:08.410869 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:08.410869 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:08.410869 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:08.410869 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:37:08.410869 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:08.410869 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:37:08.410869 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:37:08.410869 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:08.410869 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:37:08.410869 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:08.410869 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:37:08.410869 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:37:08.411402 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:08.411402 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:37:08.411402 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:08.411402 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:37:08.415265 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:37:08.415663 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:37:08.415814 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:37:08.416014 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:37:08.416434 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:37:08.416688 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:37:08.416873 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:08.417452 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:37:08.417452 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:08.417452 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:37:08.417452 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:37:08.417452 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:08.417452 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:37:08.417452 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:08.417452 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:37:08.418308 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:08.418308 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:37:08.418308 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:08.418308 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:37:08.418308 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:37:08.418308 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:37:08.418308 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:37:08.418308 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:08.418308 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:08.418308 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:08.418308 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:37:08.418308 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:08.418674 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:08.418674 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:08.418674 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:37:08.418674 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:08.418674 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:37:08.418674 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:37:08.418674 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:08.418674 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:37:08.418674 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:08.418674 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:37:08.418674 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:37:08.418674 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:08.418674 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:37:08.419042 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:08.419042 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:37:08.419394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:08.419394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:37:08.419394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:37:08.419394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:37:08.419394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:37:08.419394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:37:08.419394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:08.419394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:37:08.419394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:37:08.419394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:08.419394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:37:08.419394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:37:08.419394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:08.419394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:08.419394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:37:08.419394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:08.419830 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:37:08.419830 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:37:08.419830 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:08.419830 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:08.419830 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:08.419830 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:37:08.419830 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:08.419830 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:08.419830 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:08.419830 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:37:08.419830 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:08.419830 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:37:08.419830 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:37:08.419830 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:08.419830 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:37:08.419830 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:08.419830 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:37:08.419830 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:37:08.419830 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:08.420338 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:37:08.420338 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:08.420338 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:37:12.927381 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:37:12.928181 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:37:12.928375 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:37:12.929826 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:37:12.968624 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:37:12.969104 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:12.970172 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:37:12.970172 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:12.970172 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:37:12.970172 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:37:12.970172 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:12.970172 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:37:12.970172 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:12.970172 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:37:12.971237 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.042 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:12.971237 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:37:12.971237 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:12.971237 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:37:12.971237 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:37:12.971237 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:37:12.971237 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:37:12.971237 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:12.971237 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:12.971237 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:12.971237 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:37:12.971237 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:12.972064 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:12.972064 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:12.972064 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:37:12.972064 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:12.972064 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:37:12.972064 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:37:12.972064 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:12.972064 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:37:12.972064 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:12.972064 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:37:12.972064 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:37:12.972064 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:12.972064 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:37:12.972890 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:12.972890 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:37:12.972890 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:12.972890 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:37:12.972890 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:37:12.972890 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:37:12.972890 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:37:12.972890 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:37:12.972890 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:12.972890 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:37:12.972890 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:37:12.972890 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:12.972890 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:37:12.972890 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:37:12.972890 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:12.973879 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:12.973879 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:37:12.973879 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:12.973879 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:37:12.973879 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:37:12.973879 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:12.973879 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:12.973879 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:12.973879 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:37:12.973879 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:12.973879 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:12.973879 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:12.973879 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:37:12.973879 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:12.973879 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:37:12.973879 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:37:12.973879 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:12.973879 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:37:12.973879 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:12.975075 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:37:12.975075 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:37:12.975075 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:12.975075 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:37:12.975075 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:12.975075 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:37:13.391490 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:37:13.392281 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:37:13.392544 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:37:13.393589 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:37:13.393900 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:13.394756 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:37:13.394756 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:13.394756 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:37:13.394756 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:37:13.394756 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:13.394756 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:37:13.394756 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:13.394756 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:37:13.396375 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:13.396375 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:37:13.396375 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:13.396375 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:37:13.396375 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:37:13.396375 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:37:13.396375 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:37:13.396375 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:13.396375 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:13.396375 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:13.396375 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:37:13.396375 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:13.396951 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:13.396951 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:13.396951 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:37:13.396951 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:13.396951 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:37:13.396951 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:37:13.396951 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:13.396951 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:37:13.396951 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:13.396951 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:37:13.396951 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:37:13.396951 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:13.396951 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:37:13.397579 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:13.397579 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:37:13.398133 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:13.398133 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:37:13.398133 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:37:13.398133 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:37:13.398133 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:37:13.398133 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:37:13.398133 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:13.398133 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:37:13.398133 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:37:13.398133 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:13.398133 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:37:13.398133 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:37:13.398133 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:13.398133 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:13.398133 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:37:13.398133 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:13.399181 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:37:13.399181 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:37:13.399181 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:13.399181 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:13.399181 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:13.399181 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:37:13.399181 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:13.399181 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:13.399181 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:13.399181 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:37:13.399181 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:13.399181 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:37:13.399181 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:37:13.399181 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:13.399181 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:37:13.399181 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:13.399181 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:37:13.399181 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:37:13.399181 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:13.400460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:37:13.400460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:13.400460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:37:13.400460 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:37:13.400460 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:37:13.400460 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:37:13.400460 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:37:13.400814 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:37:13.400959 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:13.401705 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:37:13.401705 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:13.401705 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:37:13.401705 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:37:13.401705 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:13.401705 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:37:13.401705 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:13.401705 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:37:13.402705 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:13.402705 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:37:13.402705 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:37:13.402705 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:37:13.402705 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:13.402705 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:37:13.402705 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:13.402705 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:37:13.402705 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:13.402705 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:37:13.402705 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:13.402705 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:37:13.402705 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:13.402705 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:37:13.402705 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:37:13.403394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:13.403394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:37:13.403394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:13.403394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:37:13.403394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:37:13.403394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:13.403394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:37:13.403394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:13.403394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:37:13.403954 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:13.403954 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:37:13.403954 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:13.403954 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:37:13.403954 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:37:13.403954 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:37:13.403954 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:37:13.403954 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:37:13.403954 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:37:13.403954 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:37:13.403954 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:13.403954 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:13.404562 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:13.404562 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:37:13.404562 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:13.404562 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:13.404562 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:13.404562 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:37:13.404562 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:13.404562 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:37:13.404562 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:37:13.404562 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:13.404562 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:37:13.404562 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:13.404562 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:37:13.404562 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:37:13.405118 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:13.405118 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:37:13.405118 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:13.405118 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:37:13.405705 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:13.405705 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:37:13.405705 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:37:13.405705 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:37:13.405705 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:37:13.405705 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:37:13.405705 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:13.405705 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:37:13.405705 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:37:13.405705 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:13.405705 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:37:13.405705 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:37:13.405705 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:13.405705 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:13.405705 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:37:13.405705 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:13.406242 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:37:13.406242 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:37:13.406242 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:37:13.406242 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:37:13.406242 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:37:13.406242 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:13.406242 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:13.406242 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:13.406242 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:37:13.406242 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:13.406242 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:13.406242 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:13.406242 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:37:13.406242 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:13.406242 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:37:13.406242 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:37:13.406242 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:13.406242 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:37:13.407101 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:13.407101 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:37:13.407101 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:37:13.407101 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:13.407101 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:37:13.407101 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:13.407101 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:37:13.407101 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:37:13.407101 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:37:13.407101 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:37:13.407440 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:37:13.407440 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:37:13.407740 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:13.408500 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:37:13.408500 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:13.408500 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:37:13.408500 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:37:13.408500 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:13.408500 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:37:13.408500 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:13.408500 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:37:13.409432 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:13.409432 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:37:13.409432 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:37:13.409432 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:37:13.409432 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:13.409432 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:37:13.409432 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:13.409432 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:37:13.409432 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:13.409432 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:37:13.409432 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:13.409432 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:37:13.409432 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:13.409432 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:37:13.409432 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:37:13.410057 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:13.410057 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:37:13.410057 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:13.410057 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:37:13.410057 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:37:13.410057 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:13.410057 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:37:13.410057 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:13.410057 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:37:13.410708 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:13.410708 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:37:13.410708 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:13.410708 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:37:13.410708 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:37:13.410708 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:37:13.410708 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:37:13.410708 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:37:13.410708 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:37:13.410708 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:13.410708 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:13.411216 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:13.411216 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:37:13.411216 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:13.411216 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:13.411216 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:13.411216 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:37:13.411216 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:13.411216 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:37:13.411216 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:37:13.411216 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:13.411216 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:37:13.411216 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:13.411216 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:37:13.411216 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:37:13.411792 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:13.411792 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:37:13.411792 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:13.411792 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:37:13.412306 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:13.412306 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:37:13.412306 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:37:13.412306 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:37:13.412306 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:37:13.412306 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:37:13.412306 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:13.412306 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:37:13.412306 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:37:13.412306 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:13.412306 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:37:13.412306 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:37:13.412306 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:13.412306 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:13.412306 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:37:13.412306 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:13.412306 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:37:13.412908 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:37:13.412908 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:37:13.412908 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:37:13.412908 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:13.412908 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:13.412908 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:13.412908 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:37:13.412908 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:13.412908 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:13.412908 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:13.412908 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:37:13.412908 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:13.412908 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:37:13.412908 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:37:13.412908 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:13.412908 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:37:13.412908 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:13.412908 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:37:13.412908 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:37:13.413468 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:13.413468 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:37:13.413468 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:13.413468 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:37:13.415821 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:37:13.416202 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:37:13.416386 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:37:13.416599 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:37:13.417177 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:37:13.417592 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:37:13.417773 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:13.418668 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:37:13.418668 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:13.418668 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:37:13.418668 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:37:13.418668 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:13.418668 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:37:13.418668 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:13.418668 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:37:13.419739 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:13.419739 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:37:13.419739 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:13.419739 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:37:13.419739 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:37:13.419739 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:37:13.419739 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:37:13.419739 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:13.419739 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:13.419739 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:13.419739 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:37:13.419739 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:13.420334 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:13.420334 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:13.420334 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:37:13.420334 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:13.420334 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:37:13.420334 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:37:13.420334 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:13.420334 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:37:13.420334 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:13.420334 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:37:13.420334 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:37:13.420334 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:13.420334 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:37:13.420884 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:13.420884 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:37:13.421194 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:13.421194 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:37:13.421194 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:37:13.421194 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:37:13.421194 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:37:13.421194 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:37:13.421194 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:13.421194 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:37:13.421194 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:37:13.421194 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:13.421194 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:37:13.421194 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:37:13.421194 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:13.421194 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:13.421194 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:37:13.421194 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:13.421725 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:37:13.421725 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:37:13.421725 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:13.421725 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:13.421725 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:13.421725 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:37:13.421725 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:13.421725 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:13.421725 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:13.421725 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:37:13.421725 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:13.421725 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:37:13.421725 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:37:13.421725 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:13.421725 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:37:13.421725 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:13.421725 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:37:13.421725 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:37:13.421725 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:13.422431 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:37:13.422431 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:13.422431 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:37:17.928371 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:37:17.929703 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:37:17.929916 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:37:17.931501 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:37:17.968689 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:37:17.969191 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:17.970484 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:37:17.970484 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:17.970484 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:37:17.970484 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:37:17.970484 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:17.970484 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:37:17.970484 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:17.970484 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:37:17.972210 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.041 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:17.972210 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:37:17.972210 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:17.972210 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:37:17.972210 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:37:17.972210 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:37:17.972210 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:37:17.972210 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:17.972210 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:17.972210 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:17.972210 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:37:17.972210 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:17.972811 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:17.972811 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:17.972811 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:37:17.972811 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:17.972811 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:37:17.972811 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:37:17.972811 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:17.972811 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:37:17.972811 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:17.972811 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:37:17.972811 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:37:17.972811 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:17.972811 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:37:17.973429 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:17.973429 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:37:17.974401 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:17.974401 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:37:17.974401 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:37:17.974401 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:37:17.974401 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:37:17.974401 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:37:17.974401 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:17.974401 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:37:17.974401 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:37:17.974401 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:17.974401 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:37:17.974401 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:37:17.974401 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:17.974401 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:17.974401 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:37:17.974401 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:17.975120 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:37:17.975120 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:37:17.975120 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:17.975120 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:17.975120 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:17.975120 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:37:17.975120 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:17.975120 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:17.975120 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:17.975120 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:37:17.975120 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:17.975120 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:37:17.975120 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:37:17.975120 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:17.975120 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:37:17.975120 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:17.975120 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:37:17.975120 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:37:17.975120 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:17.975977 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:37:17.975977 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:17.975977 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:37:18.392403 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:37:18.393229 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:37:18.393444 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:37:18.394794 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:37:18.395204 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:18.396253 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:37:18.396253 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:18.396253 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:37:18.396253 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:37:18.396253 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:18.396253 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:37:18.396253 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:18.396253 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:37:18.398031 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:18.398031 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:37:18.398031 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:18.398031 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:37:18.398031 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:37:18.398031 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:37:18.398031 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:37:18.398031 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:18.398031 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:18.398031 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:18.398031 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:37:18.398031 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:18.398717 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:18.398717 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:18.398717 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:37:18.398717 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:18.398717 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:37:18.398717 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:37:18.398717 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:18.398717 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:37:18.398717 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:18.398717 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:37:18.398717 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:37:18.398717 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:18.398717 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:37:18.399503 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:18.399503 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:37:18.400310 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:18.400310 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:37:18.400310 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:37:18.400310 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:37:18.400310 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:37:18.400310 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:37:18.400310 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:18.400310 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:37:18.400310 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:37:18.400310 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:18.400310 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:37:18.400310 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:37:18.400310 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:18.400310 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:18.400310 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:37:18.400310 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:18.401050 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:37:18.401050 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:37:18.401050 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:18.401050 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:18.401050 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:18.401050 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:37:18.401050 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:18.401050 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:18.401050 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:18.401050 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:37:18.401050 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:18.401050 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:37:18.401050 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:37:18.401050 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:18.401050 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:37:18.401050 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:18.401050 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:37:18.401050 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:37:18.401050 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:18.401855 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:37:18.401855 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:18.401855 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:37:18.401855 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:37:18.402061 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:37:18.402061 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:37:18.402193 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:37:18.402570 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:37:18.402789 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:18.403430 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:37:18.403430 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:18.403430 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:37:18.403430 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:37:18.403430 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:18.403430 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:37:18.403430 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:18.403430 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:37:18.404232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:18.404232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:37:18.404232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:37:18.404232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:37:18.404232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:18.404232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:37:18.404232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:18.404232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:37:18.404232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:18.404232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:37:18.404232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:18.404232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:37:18.404232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:18.404232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:37:18.404232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:37:18.404713 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:18.404713 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:37:18.404713 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:18.404713 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:37:18.404713 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:37:18.404713 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:18.404713 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:37:18.404713 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:18.404713 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:37:18.405225 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:18.405225 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:37:18.405225 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:18.405225 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:37:18.405225 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:37:18.405225 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:37:18.405225 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:37:18.405225 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:37:18.405225 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:37:18.405225 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:37:18.405225 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:18.405225 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:18.405629 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:18.405629 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:37:18.405629 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:18.405629 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:18.405629 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:18.405629 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:37:18.405629 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:18.405629 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:37:18.405629 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:37:18.405629 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:18.405629 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:37:18.405629 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:18.405629 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:37:18.405629 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:37:18.405988 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:18.405988 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:37:18.405988 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:18.405988 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:37:18.406237 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:18.406237 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:37:18.406237 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:37:18.406237 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:37:18.406237 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:37:18.406237 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:37:18.406237 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:18.406237 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:37:18.406237 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:37:18.406237 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:18.406237 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:37:18.406237 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:37:18.406237 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:18.406237 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:18.406237 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:37:18.406237 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:18.406695 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:37:18.406695 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:37:18.406695 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:37:18.406695 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:37:18.406695 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:37:18.406695 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:18.406695 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:18.406695 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:18.406695 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:37:18.406695 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:18.406695 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:18.406695 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:18.406695 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:37:18.406695 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:18.406695 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:37:18.406695 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:37:18.406695 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:18.406695 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:37:18.407176 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:18.407176 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:37:18.407176 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:37:18.407176 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:18.407176 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:37:18.407176 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:18.407176 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:37:18.407176 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:37:18.407176 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:37:18.407176 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:37:18.407486 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:37:18.408236 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:37:18.408528 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:18.409131 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:37:18.409131 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:18.409131 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:37:18.409131 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:37:18.409131 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:18.409131 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:37:18.409131 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:18.409131 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:37:18.409897 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:18.409897 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:37:18.409897 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:37:18.409897 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:37:18.409897 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:18.409897 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:37:18.409897 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:18.409897 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:37:18.409897 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:18.409897 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:37:18.409897 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:18.409897 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:37:18.409897 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:18.409897 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:37:18.409897 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:37:18.410357 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:18.410357 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:37:18.410357 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:18.410357 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:37:18.410357 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:37:18.410357 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:18.410357 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:37:18.410357 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:18.410357 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:37:18.410780 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:18.410780 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:37:18.410780 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:18.410780 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:37:18.410780 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:37:18.410780 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:37:18.410780 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:37:18.410780 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:37:18.410780 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:37:18.410780 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:18.410780 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:18.411137 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:18.411137 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:37:18.411137 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:18.411137 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:18.411137 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:18.411137 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:37:18.411137 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:18.411137 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:37:18.411137 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:37:18.411137 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:18.411137 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:37:18.411137 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:18.411137 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:37:18.411137 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:37:18.411509 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:18.411509 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:37:18.411509 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:18.411509 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:37:18.411818 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:18.411818 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:37:18.411818 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:37:18.411818 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:37:18.411818 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:37:18.411818 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:37:18.411818 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:18.411818 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:37:18.411818 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:37:18.411818 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:18.411818 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:37:18.411818 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:37:18.411818 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:18.411818 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:18.411818 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:37:18.411818 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:18.411818 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:37:18.412378 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:37:18.412378 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:37:18.412378 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:37:18.412378 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:18.412378 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:18.412378 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:18.412378 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:37:18.412378 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:18.412378 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:18.412378 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:18.412378 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:37:18.412378 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:18.412378 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:37:18.412378 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:37:18.412378 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:18.412378 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:37:18.412378 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:18.412378 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:37:18.412378 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:37:18.413349 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:18.413349 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:37:18.413349 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:18.413349 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:37:18.416231 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:37:18.416712 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:37:18.416942 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:37:18.417214 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:37:18.417822 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:37:18.418246 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:37:18.418526 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:18.419350 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:37:18.419350 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:18.419350 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:37:18.419350 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:37:18.419350 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:18.419350 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:37:18.419350 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:18.419350 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:37:18.420189 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:18.420189 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:37:18.420189 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:18.420189 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:37:18.420189 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:37:18.420189 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:37:18.420189 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:37:18.420189 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:18.420189 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:18.420189 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:18.420189 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:37:18.420189 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:18.420654 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:18.420654 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:18.420654 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:37:18.420654 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:18.420654 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:37:18.420654 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:37:18.420654 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:18.420654 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:37:18.420654 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:18.420654 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:37:18.420654 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:37:18.420654 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:18.420654 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:37:18.421015 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:18.421015 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:37:18.421189 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:18.421189 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:37:18.421189 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:37:18.421189 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:37:18.421189 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:37:18.421189 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:37:18.421189 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:18.421189 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:37:18.421189 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:37:18.421189 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:18.421189 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:37:18.421189 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:37:18.421189 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:18.421189 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:18.421189 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:37:18.421189 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:18.421652 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:37:18.421652 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:37:18.421652 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:18.421652 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:18.421652 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:18.421652 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:37:18.421652 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:18.421652 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:18.421652 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:18.421652 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:37:18.421652 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:18.421652 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:37:18.421652 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:37:18.421652 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:18.421652 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:37:18.421652 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:18.421652 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:37:18.421652 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:37:18.421652 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:18.422206 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:37:18.422206 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:18.422206 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:37:22.929235 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:37:22.930192 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:37:22.930192 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:37:22.931273 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:37:22.967502 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:37:22.968053 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:22.969138 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:37:22.969138 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:22.969138 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:37:22.969138 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:37:22.969138 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:22.969138 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:37:22.969138 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:22.969138 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:37:22.970359 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.039 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:22.970359 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:37:22.970359 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:22.970359 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:37:22.970359 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:37:22.970359 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:37:22.970359 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:37:22.970359 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:22.970359 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:22.970359 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:22.970359 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:37:22.970359 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:22.971067 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:22.971067 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:22.971067 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:37:22.971067 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:22.971067 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:37:22.971067 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:37:22.971067 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:22.971067 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:37:22.971067 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:22.971067 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:37:22.971067 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:37:22.971067 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:22.971067 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:37:22.971871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:22.971871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:37:22.971871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:22.971871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:37:22.971871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:37:22.971871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:37:22.971871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:37:22.971871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:37:22.971871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:22.971871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:37:22.971871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:37:22.971871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:22.971871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:37:22.971871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:37:22.971871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:22.972806 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:22.972806 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:37:22.972806 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:22.972806 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:37:22.972806 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:37:22.972806 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:22.972806 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:22.972806 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:22.972806 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:37:22.972806 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:22.972806 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:22.972806 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:22.972806 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:37:22.972806 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:22.972806 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:37:22.972806 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:37:22.972806 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:22.972806 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:37:22.972806 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:22.973341 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:37:22.973341 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:37:22.973341 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:22.973341 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:37:22.973341 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:22.973341 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:37:23.393277 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:37:23.394024 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:37:23.394216 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:37:23.395372 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:37:23.395836 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:23.396827 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:37:23.396827 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:23.396827 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:37:23.396827 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:37:23.396827 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:23.396827 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:37:23.396827 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:23.396827 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:37:23.398503 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:23.398503 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:37:23.398503 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:23.398503 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:37:23.398503 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:37:23.398503 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:37:23.398503 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:37:23.398503 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:23.398503 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:23.398503 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:23.398503 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:37:23.398503 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:23.399315 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:23.399315 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:23.399315 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:37:23.399315 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:23.399315 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:37:23.399315 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:37:23.399315 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:23.399315 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:37:23.399315 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:23.399315 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:37:23.399315 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:37:23.399315 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:23.399315 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:37:23.399980 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:23.399980 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:37:23.400640 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:23.400640 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:37:23.400640 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:37:23.400640 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:37:23.400640 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:37:23.400640 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:37:23.400640 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:23.400640 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:37:23.400640 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:37:23.400640 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:23.400640 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:37:23.400640 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:37:23.400640 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:23.400640 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:23.400640 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:37:23.400640 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:23.401476 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:37:23.401476 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:37:23.401476 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:23.401476 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:23.401476 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:23.401476 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:37:23.401476 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:23.401476 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:23.401476 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:23.401476 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:37:23.401476 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:23.401476 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:37:23.401476 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:37:23.401476 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:23.401476 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:37:23.401476 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:23.401476 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:37:23.401476 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:37:23.401476 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:23.402497 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:37:23.402497 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:23.402497 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:37:23.402497 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:37:23.402497 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:37:23.402801 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:37:23.403017 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:37:23.403616 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:37:23.403970 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:23.404784 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:37:23.404784 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:23.404784 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:37:23.404784 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:37:23.404784 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:23.404784 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:37:23.404784 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:23.404784 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:37:23.405567 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:23.405567 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:37:23.405567 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:37:23.405567 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:37:23.405567 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:23.405567 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:37:23.405567 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:23.405567 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:37:23.405567 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:23.405567 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:37:23.405567 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:23.405567 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:37:23.405567 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:23.405567 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:37:23.405567 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:37:23.406094 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:23.406094 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:37:23.406094 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:23.406094 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:37:23.406094 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:37:23.406094 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:23.406094 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:37:23.406094 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:23.406094 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:37:23.406613 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:23.406613 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:37:23.406613 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:23.406613 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:37:23.406613 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:37:23.406613 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:37:23.406613 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:37:23.406613 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:37:23.406613 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:37:23.406613 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:37:23.406613 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:23.406613 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:23.406968 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:23.406968 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:37:23.406968 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:23.406968 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:23.406968 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:23.406968 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:37:23.406968 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:23.406968 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:37:23.406968 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:37:23.406968 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:23.406968 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:37:23.406968 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:23.406968 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:37:23.406968 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:37:23.407386 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:23.407386 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:37:23.407386 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:23.407386 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:37:23.407825 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:23.407825 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:37:23.407825 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:37:23.407825 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:37:23.407825 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:37:23.407825 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:37:23.407825 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:23.407825 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:37:23.407825 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:37:23.407825 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:23.407825 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:37:23.407825 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:37:23.407825 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:23.407825 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:23.407825 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:37:23.407825 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:23.408288 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:37:23.408288 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:37:23.408288 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:37:23.408288 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:37:23.408288 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:37:23.408288 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:23.408288 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:23.408288 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:23.408288 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:37:23.408288 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:23.408288 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:23.408288 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:23.408288 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:37:23.408288 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:23.408288 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:37:23.408288 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:37:23.408288 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:23.408288 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:37:23.408875 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:23.408875 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:37:23.408875 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:37:23.408875 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:23.408875 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:37:23.408875 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:23.408875 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:37:23.408875 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:37:23.408875 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:37:23.408875 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:37:23.409216 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:37:23.409446 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:37:23.409891 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:23.410793 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:37:23.410793 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:23.410793 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:37:23.410793 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:37:23.410793 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:23.410793 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:37:23.410793 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:23.410793 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:37:23.411586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:23.411586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:37:23.411586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:37:23.411586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:37:23.411586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:23.411586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:37:23.411586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:23.411586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:37:23.411586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:23.411586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:37:23.411586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:23.411586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:37:23.411586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:23.411586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:37:23.411586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:37:23.412152 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:23.412152 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:37:23.412152 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:23.412152 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:37:23.412152 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:37:23.412152 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:23.412152 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:37:23.412152 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:23.412152 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:37:23.412622 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:23.412622 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:37:23.412622 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:23.412622 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:37:23.412622 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:37:23.412622 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:37:23.412622 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:37:23.412622 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:37:23.412622 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:37:23.412622 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:23.412622 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:23.412942 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:23.412942 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:37:23.412942 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:23.412942 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:23.412942 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:23.412942 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:37:23.412942 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:23.412942 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:37:23.412942 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:37:23.412942 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:23.412942 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:37:23.412942 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:23.412942 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:37:23.412942 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:37:23.413309 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:23.413309 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:37:23.413309 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:23.413309 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:37:23.413659 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:23.413659 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:37:23.413659 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:37:23.413659 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:37:23.413659 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:37:23.413659 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:37:23.413659 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:23.413659 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:37:23.413659 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:37:23.413659 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:23.413659 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:37:23.413659 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:37:23.413659 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:23.413659 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:23.413659 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:37:23.413659 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:23.413659 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:37:23.414154 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:37:23.414154 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:37:23.414154 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:37:23.414154 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:23.414154 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:23.414154 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:23.414154 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:37:23.414154 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:23.414154 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:23.414154 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:23.414154 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:37:23.414154 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:23.414154 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:37:23.414154 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:37:23.414154 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:23.414154 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:37:23.414154 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:23.414154 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:37:23.414154 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:37:23.414648 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:23.414648 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:37:23.414648 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:23.414648 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:37:23.416789 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:37:23.417122 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:37:23.417220 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:37:23.417398 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:37:23.417892 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:37:23.418292 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:37:23.418552 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:23.419432 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:37:23.419432 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:23.419432 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:37:23.419432 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:37:23.419432 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:23.419432 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:37:23.419432 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:23.419432 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:37:23.420236 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:23.420236 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:37:23.420236 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:23.420236 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:37:23.420236 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:37:23.420236 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:37:23.420236 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:37:23.420236 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:23.420236 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:23.420236 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:23.420236 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:37:23.420236 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:23.420594 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:23.420594 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:23.420594 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:37:23.420594 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:23.420594 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:37:23.420594 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:37:23.420594 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:23.420594 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:37:23.420594 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:23.420594 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:37:23.420594 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:37:23.420594 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:23.420594 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:37:23.420925 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:23.420925 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:37:23.421172 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:23.421172 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:37:23.421172 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:37:23.421172 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:37:23.421172 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:37:23.421172 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:37:23.421172 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:23.421172 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:37:23.421172 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:37:23.421172 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:23.421172 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:37:23.421172 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:37:23.421172 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:23.421172 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:23.421172 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:37:23.421172 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:23.422029 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:37:23.422029 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:37:23.422029 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:23.422029 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:23.422029 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:23.422029 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:37:23.422029 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:23.422029 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:23.422029 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:23.422029 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:37:23.422029 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:23.422029 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:37:23.422029 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:37:23.422029 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:23.422029 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:37:23.422029 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:23.422029 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:37:23.422029 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:37:23.422029 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:23.422554 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:37:23.422554 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:23.422554 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:37:27.930097 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:37:27.931048 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:37:27.931048 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:37:27.932198 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:37:27.970006 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:37:27.970278 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:27.970964 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:37:27.970964 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:27.970964 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:37:27.970964 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:37:27.970964 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:27.970964 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:37:27.970964 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:27.970964 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:37:27.971714 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.040 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:27.971714 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:37:27.971714 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:27.971714 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:37:27.971714 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:37:27.971714 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:37:27.971714 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:37:27.971714 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:27.971714 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:27.971714 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:27.971714 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:37:27.971714 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:27.972916 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:27.972916 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:27.972916 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:37:27.972916 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:27.972916 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:37:27.972916 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:37:27.972916 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:27.972916 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:37:27.972916 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:27.972916 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:37:27.972916 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:37:27.972916 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:27.972916 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:37:27.973459 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:27.973459 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:37:27.973459 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:27.973459 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:37:27.973459 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:37:27.973459 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:37:27.973459 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:37:27.973459 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:37:27.973459 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:27.973459 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:37:27.973459 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:37:27.973459 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:27.973459 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:37:27.973459 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:37:27.973459 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:27.974736 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:27.974736 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:37:27.974736 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:27.974736 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:37:27.974736 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:37:27.974736 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:27.974736 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:27.974736 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:27.974736 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:37:27.974736 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:27.974736 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:27.974736 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:27.974736 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:37:27.974736 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:27.974736 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:37:27.974736 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:37:27.974736 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:27.974736 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:37:27.974736 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:27.976704 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:37:27.976704 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:37:27.976704 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:27.976704 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:37:27.976704 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:27.976704 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:37:28.393941 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:37:28.394564 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:37:28.394843 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:37:28.396134 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:37:28.396527 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:28.397455 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:37:28.397455 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:28.397455 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:37:28.397455 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:37:28.397455 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:28.397455 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:37:28.397455 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:28.397455 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:37:28.398622 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:28.398622 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:37:28.398622 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:28.398622 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:37:28.398622 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:37:28.398622 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:37:28.398622 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:37:28.398622 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:28.398622 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:28.398622 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:28.398622 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:37:28.398622 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:28.399372 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:28.399372 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:28.399372 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:37:28.399372 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:28.399372 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:37:28.399372 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:37:28.399372 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:28.399372 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:37:28.399372 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:28.399372 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:37:28.399372 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:37:28.399372 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:28.399372 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:37:28.400290 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:28.400290 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:37:28.400290 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:28.400290 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:37:28.400290 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:37:28.400290 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:37:28.400290 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:37:28.400290 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:37:28.400290 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:28.400290 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:37:28.400290 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:37:28.400290 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:28.400290 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:37:28.400290 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:37:28.400290 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:28.401363 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:28.401363 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:37:28.401363 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:28.401363 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:37:28.401363 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:37:28.401363 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:28.401363 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:28.401363 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:28.401363 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:37:28.401363 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:28.401363 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:28.401363 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:28.401363 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:37:28.401363 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:28.401363 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:37:28.401363 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:37:28.401363 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:28.401363 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:37:28.401363 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:28.402205 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:37:28.402205 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:37:28.402205 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:28.402205 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:37:28.402205 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:28.402205 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:37:28.402607 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:37:28.402899 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:37:28.403018 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:37:28.403147 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:37:28.403695 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:37:28.404015 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:28.404756 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:37:28.404756 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:28.404756 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:37:28.404756 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:37:28.404756 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:28.404756 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:37:28.404756 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:28.404756 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:37:28.405605 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:28.405605 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:37:28.405605 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:37:28.405605 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:37:28.405605 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:28.405605 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:37:28.405605 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:28.405605 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:37:28.405605 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:28.405605 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:37:28.405605 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:28.405605 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:37:28.405605 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:28.405605 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:37:28.405605 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:37:28.406139 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:28.406139 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:37:28.406139 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:28.406139 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:37:28.406139 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:37:28.406139 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:28.406139 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:37:28.406139 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:28.406139 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:37:28.406511 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:28.406511 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:37:28.406511 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:28.406511 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:37:28.406511 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:37:28.406511 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:37:28.406511 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:37:28.406511 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:37:28.406511 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:37:28.406511 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:37:28.406511 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:28.406511 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:28.406930 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:28.406930 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:37:28.406930 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:28.406930 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:28.406930 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:28.406930 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:37:28.406930 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:28.406930 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:37:28.406930 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:37:28.406930 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:28.406930 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:37:28.406930 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:28.406930 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:37:28.406930 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:37:28.407459 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:28.407459 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:37:28.407459 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:28.407459 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:37:28.407602 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:28.407602 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:37:28.407602 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:37:28.407602 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:37:28.407602 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:37:28.407602 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:37:28.407602 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:28.407602 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:37:28.407602 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:37:28.407602 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:28.407602 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:37:28.407602 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:37:28.407602 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:28.407602 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:28.407602 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:37:28.407602 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:28.408234 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:37:28.408234 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:37:28.408234 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:37:28.408234 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:37:28.408234 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:37:28.408234 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:28.408234 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:28.408234 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:28.408234 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:37:28.408234 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:28.408234 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:28.408234 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:28.408234 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:37:28.408234 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:28.408234 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:37:28.408234 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:37:28.408234 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:28.408234 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:37:28.408923 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:28.408923 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:37:28.408923 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:37:28.408923 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:28.408923 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:37:28.408923 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:28.408923 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:37:28.408923 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:37:28.408923 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:37:28.408923 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:37:28.409409 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:37:28.409587 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:37:28.409941 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:28.410749 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:37:28.410749 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:28.410749 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:37:28.410749 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:37:28.410749 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:28.410749 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:37:28.410749 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:28.410749 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:37:28.411569 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:28.411569 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:37:28.411569 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:37:28.411569 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:37:28.411569 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:28.411569 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:37:28.411569 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:28.411569 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:37:28.411569 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:28.411569 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:37:28.411569 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:28.411569 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:37:28.411569 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:28.411569 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:37:28.411569 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:37:28.412097 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:28.412097 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:37:28.412097 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:28.412097 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:37:28.412097 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:37:28.412097 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:28.412097 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:37:28.412097 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:28.412097 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:37:28.412470 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:28.412470 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:37:28.412470 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:28.412470 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:37:28.412470 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:37:28.412470 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:37:28.412470 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:37:28.412470 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:37:28.412470 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:37:28.412470 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:28.412470 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:28.412932 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:28.412932 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:37:28.412932 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:28.412932 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:28.412932 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:28.412932 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:37:28.412932 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:28.412932 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:37:28.412932 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:37:28.412932 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:28.412932 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:37:28.412932 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:28.412932 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:37:28.412932 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:37:28.413511 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:28.413511 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:37:28.413511 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:28.413511 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:37:28.413675 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:28.413675 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:37:28.413675 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:37:28.413675 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:37:28.413675 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:37:28.413675 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:37:28.413675 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:28.413675 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:37:28.413675 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:37:28.413675 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:28.413675 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:37:28.413675 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:37:28.413675 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:28.413675 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:28.413675 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:37:28.413675 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:28.413675 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:37:28.414374 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:37:28.414374 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:37:28.414374 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:37:28.414374 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:28.414374 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:28.414374 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:28.414374 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:37:28.414374 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:28.414374 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:28.414374 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:28.414374 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:37:28.414374 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:28.414374 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:37:28.414374 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:37:28.414374 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:28.414374 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:37:28.414374 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:28.414374 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:37:28.414374 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:37:28.415110 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:28.415110 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:37:28.415110 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:28.415110 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:37:28.417336 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:37:28.417659 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:37:28.417773 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:37:28.417932 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:37:28.418422 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:37:28.418762 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:37:28.418946 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:28.419684 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:37:28.419684 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:28.419684 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:37:28.419684 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:37:28.419684 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:28.419684 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:37:28.419684 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:28.419684 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:37:28.420557 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:28.420557 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:37:28.420557 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:28.420557 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:37:28.420557 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:37:28.420557 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:37:28.420557 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:37:28.420557 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:28.420557 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:28.420557 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:28.420557 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:37:28.420557 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:28.421059 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:28.421059 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:28.421059 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:37:28.421059 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:28.421059 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:37:28.421059 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:37:28.421059 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:28.421059 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:37:28.421059 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:28.421059 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:37:28.421059 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:37:28.421059 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:28.421059 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:37:28.421628 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:28.421628 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:37:28.421628 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:28.421628 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:37:28.421628 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:37:28.421628 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:37:28.421628 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:37:28.421628 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:37:28.421628 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:28.421628 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:37:28.421628 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:37:28.421628 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:28.421628 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:37:28.421628 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:37:28.421628 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:28.422257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:28.422257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:37:28.422257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:28.422257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:37:28.422257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:37:28.422257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:28.422257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:28.422257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:28.422257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:37:28.422257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:28.422257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:28.422257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:28.422257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:37:28.422257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:28.422257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:37:28.422257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:37:28.422257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:28.422257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:37:28.422257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:28.423016 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:37:28.423016 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:37:28.423016 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:28.423016 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:37:28.423016 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:28.423016 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:37:32.930932 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:37:32.931804 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:37:32.931804 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:37:32.933171 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:37:32.964851 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:37:32.965297 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:32.966260 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:37:32.966260 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:32.966260 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:37:32.966260 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:37:32.966260 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:32.966260 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:37:32.966260 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:32.966260 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:37:32.967085 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.034 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:32.967085 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:37:32.967085 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:32.967085 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:37:32.967085 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:37:32.967085 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:37:32.967085 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:37:32.967085 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:32.967085 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:32.967085 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:32.967085 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:37:32.967085 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:32.967686 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:32.967686 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:32.967686 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:37:32.967686 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:32.967686 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:37:32.967686 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:37:32.967686 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:32.967686 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:37:32.967686 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:32.967686 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:37:32.967686 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:37:32.967686 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:32.967686 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:37:32.968401 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:32.968401 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:37:32.968401 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:32.968401 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:37:32.968401 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:37:32.968401 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:37:32.968401 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:37:32.968401 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:37:32.968401 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:32.968401 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:37:32.968401 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:37:32.968401 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:32.968401 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:37:32.968401 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:37:32.968401 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:32.969078 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:32.969078 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:37:32.969078 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:32.969078 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:37:32.969078 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:37:32.969078 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:32.969078 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:32.969078 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:32.969078 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:37:32.969078 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:32.969078 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:32.969078 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:32.969078 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:37:32.969078 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:32.969078 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:37:32.969078 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:37:32.969078 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:32.969078 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:37:32.969078 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:32.970054 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:37:32.970054 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:37:32.970054 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:32.970054 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:37:32.970054 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:32.970054 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:37:33.394872 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:37:33.395772 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:37:33.396032 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:37:33.397297 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:37:33.397756 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:33.398783 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:37:33.398783 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:33.398783 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:37:33.398783 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:37:33.398783 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:33.398783 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:37:33.398783 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:33.398783 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:37:33.400588 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:33.400588 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:37:33.400588 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:33.400588 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:37:33.400588 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:37:33.400588 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:37:33.400588 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:37:33.400588 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:33.400588 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:33.400588 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:33.400588 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:37:33.400588 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:33.401399 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:33.401399 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:33.401399 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:37:33.401399 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:33.401399 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:37:33.401399 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:37:33.401399 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:33.401399 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:37:33.401399 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:33.401399 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:37:33.401399 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:37:33.401399 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:33.401399 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:37:33.402167 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:33.402167 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:37:33.402310 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:33.402310 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:37:33.402310 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:37:33.402310 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:37:33.402310 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:37:33.402310 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:37:33.402310 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:33.402310 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:37:33.402310 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:37:33.402310 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:33.402310 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:37:33.402310 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:37:33.402310 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:33.402310 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:33.402310 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:37:33.402310 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:33.403272 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:37:33.403272 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:37:33.403272 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:33.403272 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:33.403272 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:33.403272 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:37:33.403272 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:33.403272 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:33.403272 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:33.403272 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:37:33.403272 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:33.403272 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:37:33.403272 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:37:33.403272 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:33.403272 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:37:33.403272 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:33.403272 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:37:33.403272 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:37:33.403272 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:33.404526 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:37:33.404526 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:33.404526 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:37:33.404526 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:37:33.404526 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:37:33.404526 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:37:33.404526 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:37:33.404913 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:37:33.404990 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:33.405991 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:37:33.405991 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:33.405991 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:37:33.405991 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:37:33.405991 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:33.405991 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:37:33.405991 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:33.405991 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:37:33.407576 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:33.407576 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:37:33.407576 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:37:33.407576 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:37:33.407576 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:33.407576 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:37:33.407576 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:33.407576 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:37:33.407576 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:33.407576 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:37:33.407576 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:33.407576 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:37:33.407576 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:33.407576 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:37:33.407576 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:37:33.407982 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:33.407982 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:37:33.407982 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:33.407982 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:37:33.407982 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:37:33.407982 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:33.407982 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:37:33.407982 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:33.407982 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:37:33.409152 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:33.409152 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:37:33.409152 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:33.409152 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:37:33.409152 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:37:33.409152 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:37:33.409152 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:37:33.409152 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:37:33.409152 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:37:33.409152 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:37:33.409152 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:33.409152 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:33.409529 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:33.409529 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:37:33.409529 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:33.409529 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:33.409529 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:33.409529 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:37:33.409529 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:33.409529 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:37:33.409529 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:37:33.409529 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:33.409529 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:37:33.409529 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:33.409529 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:37:33.409529 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:37:33.409897 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:33.409897 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:37:33.409897 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:33.409897 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:37:33.410603 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:33.410603 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:37:33.410603 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:37:33.410603 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:37:33.410603 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:37:33.410603 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:37:33.410603 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:33.410603 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:37:33.410603 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:37:33.410603 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:33.410603 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:37:33.410603 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:37:33.410603 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:33.410603 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:33.410603 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:37:33.410603 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:33.411028 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:37:33.411028 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:37:33.411028 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:37:33.411028 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:37:33.411028 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:37:33.411028 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:33.411028 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:33.411028 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:33.411028 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:37:33.411028 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:33.411028 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:33.411028 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:33.411028 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:37:33.411028 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:33.411028 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:37:33.411028 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:37:33.411028 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:33.411028 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:37:33.411560 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:33.411560 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:37:33.411560 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:37:33.411560 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:33.411560 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:37:33.411560 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:33.411560 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:37:33.411560 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:37:33.411560 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:37:33.411560 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:37:33.411854 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:37:33.411893 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:37:33.412296 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:33.413161 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:37:33.413161 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:33.413161 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:37:33.413161 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:37:33.413161 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:33.413161 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:37:33.413161 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:33.413161 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:37:33.413925 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:33.413925 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:37:33.413925 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:37:33.413925 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:37:33.413925 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:33.413925 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:37:33.413925 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:33.413925 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:37:33.413925 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:33.413925 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:37:33.413925 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:33.413925 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:37:33.413925 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:33.413925 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:37:33.413925 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:37:33.414363 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:33.414363 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:37:33.414363 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:33.414363 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:37:33.414363 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:37:33.414363 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:33.414363 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:37:33.414363 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:33.414363 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:37:33.414871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:33.414871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:37:33.414871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:33.414871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:37:33.414871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:37:33.414871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:37:33.414871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:37:33.414871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:37:33.414871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:37:33.414871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:33.414871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:33.415190 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:33.415190 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:37:33.415190 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:33.415190 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:33.415190 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:33.415190 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:37:33.415190 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:33.415190 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:37:33.415190 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:37:33.415190 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:33.415190 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:37:33.415190 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:33.415190 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:37:33.415190 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:37:33.415608 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:33.415608 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:37:33.415608 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:33.415608 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:37:33.416072 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:33.416072 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:37:33.416072 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:37:33.416072 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:37:33.416072 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:37:33.416072 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:37:33.416072 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:33.416072 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:37:33.416072 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:37:33.416072 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:33.416072 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:37:33.416072 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:37:33.416072 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:33.416072 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:33.416072 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:37:33.416072 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:33.416072 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:37:33.416598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:37:33.416598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:37:33.416598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:37:33.416598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:33.416598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:33.416598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:33.416598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:37:33.416598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:33.416598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:33.416598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:33.416598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:37:33.416598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:33.416598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:37:33.416598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:37:33.416598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:33.416598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:37:33.416598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:33.416598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:37:33.416598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:37:33.417083 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:33.417083 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:37:33.417083 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:33.417083 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:37:33.417902 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:37:33.418272 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:37:33.418422 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:37:33.418596 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:37:33.419042 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:37:33.419443 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:37:33.419639 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:33.420503 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:37:33.420503 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:33.420503 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:37:33.420503 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:37:33.420503 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:33.420503 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:37:33.420503 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:33.420503 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:37:33.422504 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:33.422504 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:37:33.422504 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:33.422504 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:37:33.422504 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:37:33.422504 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:37:33.422504 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:37:33.422504 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:33.422504 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:33.422504 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:33.422504 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:37:33.422504 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:33.423253 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:33.423253 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:33.423253 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:37:33.423253 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:33.423253 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:37:33.423253 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:37:33.423253 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:33.423253 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:37:33.423253 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:33.423253 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:37:33.423253 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:37:33.423253 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:33.423253 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:37:33.423911 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:33.423911 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:37:33.423911 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:33.423911 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:37:33.423911 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:37:33.423911 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:37:33.423911 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:37:33.423911 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:37:33.423911 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:33.423911 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:37:33.423911 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:37:33.423911 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:33.423911 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:37:33.423911 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:37:33.423911 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:33.424710 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:33.424710 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:37:33.424710 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:33.424710 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:37:33.424710 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:37:33.424710 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:33.424710 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:33.424710 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:33.424710 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:37:33.424710 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:33.424710 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:33.424710 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:33.424710 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:37:33.424710 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:33.424710 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:37:33.424710 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:37:33.424710 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:33.424710 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:37:33.424710 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:33.425616 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:37:33.425616 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:37:33.425616 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:33.425616 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:37:33.425616 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:33.425616 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:37:37.931949 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:37:37.933021 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:37:37.933021 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:37:37.934237 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:37:37.960723 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:37:37.961149 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:37.962251 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:37:37.962251 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:37.962251 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:37:37.962251 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:37:37.962251 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:37.962251 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:37:37.962251 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:37.962251 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:37:37.963119 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.029 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:37.963119 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:37:37.963119 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:37.963119 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:37:37.963119 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:37:37.963119 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:37:37.963119 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:37:37.963119 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:37.963119 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:37.963119 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:37.963119 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:37:37.963119 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:37.963889 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:37.963889 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:37.963889 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:37:37.963889 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:37.963889 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:37:37.963889 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:37:37.963889 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:37.963889 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:37:37.963889 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:37.963889 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:37:37.963889 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:37:37.963889 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:37.963889 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:37:37.964492 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:37.964492 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:37:37.964492 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:37.964492 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:37:37.964492 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:37:37.964492 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:37:37.964492 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:37:37.964492 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:37:37.964492 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:37.964492 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:37:37.964492 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:37:37.964492 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:37.964492 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:37:37.964492 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:37:37.964492 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:37.965207 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:37.965207 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:37:37.965207 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:37.965207 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:37:37.965207 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:37:37.965207 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:37.965207 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:37.965207 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:37.965207 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:37:37.965207 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:37.965207 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:37.965207 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:37.965207 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:37:37.965207 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:37.965207 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:37:37.965207 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:37:37.965207 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:37.965207 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:37:37.965207 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:37.966087 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:37:37.966087 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:37:37.966087 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:37.966087 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:37:37.966087 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:37.966087 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:37:38.395718 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:37:38.396350 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:37:38.396590 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:37:38.397611 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:37:38.397905 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:38.398889 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:37:38.398889 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:38.398889 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:37:38.398889 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:37:38.398889 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:38.398889 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:37:38.398889 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:38.398889 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:37:38.400502 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:38.400502 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:37:38.400502 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:38.400502 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:37:38.400502 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:37:38.400502 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:37:38.400502 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:37:38.400502 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:38.400502 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:38.400502 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:38.400502 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:37:38.400502 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:38.402004 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:38.402004 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:38.402004 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:37:38.402004 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:38.402004 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:37:38.402004 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:37:38.402004 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:38.402004 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:37:38.402004 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:38.402004 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:37:38.402004 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:37:38.402004 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:38.402004 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:37:38.403281 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:38.403281 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:37:38.403281 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:38.403281 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:37:38.403281 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:37:38.403281 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:37:38.403281 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:37:38.403281 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:37:38.403281 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:38.403281 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:37:38.403281 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:37:38.403281 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:38.403281 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:37:38.403281 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:37:38.403281 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:38.404784 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:38.404784 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:37:38.404784 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:38.404784 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:37:38.404784 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:37:38.404784 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:38.404784 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:38.404784 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:38.404784 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:37:38.404784 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:38.404784 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:38.404784 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:38.404784 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:37:38.404784 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:38.404784 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:37:38.404784 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:37:38.404784 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:38.404784 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:37:38.404784 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:38.405917 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:37:38.405917 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:37:38.405917 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:38.405917 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:37:38.405917 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:38.405917 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:37:38.405917 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:37:38.405917 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:37:38.405917 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:37:38.405917 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:37:38.406519 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:37:38.406519 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:38.406519 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:37:38.406519 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:38.406519 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:37:38.406519 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:37:38.406519 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:38.406519 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:37:38.406519 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:38.406519 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:37:38.407148 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:38.407148 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:37:38.407148 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:37:38.407148 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:37:38.407148 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:38.407148 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:37:38.407148 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:38.407148 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:37:38.407148 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:38.407148 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:37:38.407148 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:38.407148 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:37:38.407148 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:38.407148 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:37:38.407148 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:37:38.407665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:38.407665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:37:38.407665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:38.407665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:37:38.407665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:37:38.407665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:38.407665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:37:38.407665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:38.407665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:37:38.407953 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:38.407953 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:37:38.407953 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:38.407953 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:37:38.407953 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:37:38.407953 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:37:38.407953 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:37:38.407953 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:37:38.407953 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:37:38.407953 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:37:38.407953 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:38.407953 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:38.408304 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:38.408304 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:37:38.408304 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:38.408304 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:38.408304 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:38.408304 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:37:38.408304 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:38.408304 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:37:38.408304 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:37:38.408304 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:38.408304 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:37:38.408304 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:38.408304 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:37:38.408304 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:37:38.408714 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:38.408714 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:37:38.408714 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:38.408714 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:37:38.408973 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:38.408973 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:37:38.408973 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:37:38.408973 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:37:38.408973 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:37:38.408973 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:37:38.408973 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:38.408973 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:37:38.408973 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:37:38.408973 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:38.408973 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:37:38.408973 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:37:38.408973 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:38.408973 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:38.408973 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:37:38.408973 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:38.409622 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:37:38.409622 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:37:38.409622 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:37:38.409622 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:37:38.409622 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:37:38.409622 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:38.409622 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:38.409622 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:38.409622 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:37:38.409622 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:38.409622 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:38.409622 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:38.409622 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:37:38.409622 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:38.409622 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:37:38.409622 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:37:38.409622 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:38.409622 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:37:38.410136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:38.410136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:37:38.410136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:37:38.410136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:38.410136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:37:38.410136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:38.410136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:37:38.411395 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:37:38.411760 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:37:38.411875 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:37:38.412298 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:37:38.412298 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:37:38.412556 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:38.413004 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:37:38.413004 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:38.413004 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:37:38.413004 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:37:38.413004 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:38.413004 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:37:38.413004 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:38.413004 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:37:38.413736 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:38.413736 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:37:38.413736 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:37:38.413736 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:37:38.413736 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:38.413736 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:37:38.413736 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:38.413736 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:37:38.413736 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:38.413736 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:37:38.413736 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:38.413736 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:37:38.413736 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:38.413736 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:37:38.413736 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:37:38.414177 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:38.414177 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:37:38.414177 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:38.414177 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:37:38.414177 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:37:38.414177 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:38.414177 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:37:38.414177 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:38.414177 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:37:38.414629 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:38.414629 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:37:38.414629 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:38.414629 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:37:38.414629 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:37:38.414629 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:37:38.414629 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:37:38.414629 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:37:38.414629 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:37:38.414629 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:38.414629 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:38.414986 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:38.414986 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:37:38.414986 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:38.414986 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:38.414986 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:38.414986 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:37:38.414986 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:38.414986 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:37:38.414986 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:37:38.414986 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:38.414986 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:37:38.414986 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:38.414986 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:37:38.414986 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:37:38.415458 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:38.415458 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:37:38.415458 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:38.415458 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:37:38.415662 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:38.415662 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:37:38.415662 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:37:38.415662 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:37:38.415662 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:37:38.415662 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:37:38.415662 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:38.415662 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:37:38.415662 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:37:38.415662 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:38.415662 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:37:38.415662 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:37:38.415662 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:38.415662 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:38.415662 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:37:38.415662 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:38.415662 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:37:38.416460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:37:38.416460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:37:38.416460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:37:38.416460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:38.416460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:38.416460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:38.416460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:37:38.416460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:38.416460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:38.416460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:38.416460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:37:38.416460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:38.416460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:37:38.416460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:37:38.416460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:38.416460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:37:38.416460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:38.416460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:37:38.416460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:37:38.417292 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:38.417292 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:37:38.417292 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:38.417292 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:37:38.418564 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:37:38.418928 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:37:38.419011 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:37:38.419177 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:37:38.419516 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:37:38.419786 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:37:38.419931 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:38.420399 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:37:38.420399 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:38.420399 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:37:38.420399 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:37:38.420399 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:38.420399 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:37:38.420399 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:38.420399 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:37:38.421252 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:38.421252 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:37:38.421252 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:38.421252 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:37:38.421252 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:37:38.421252 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:37:38.421252 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:37:38.421252 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:38.421252 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:38.421252 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:38.421252 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:37:38.421252 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:38.421761 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:38.421761 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:38.421761 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:37:38.421761 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:38.421761 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:37:38.421761 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:37:38.421761 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:38.421761 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:37:38.421761 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:38.421761 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:37:38.421761 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:37:38.421761 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:38.421761 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:37:38.422183 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:38.422183 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:37:38.422270 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:38.422270 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:37:38.422270 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:37:38.422270 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:37:38.422270 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:37:38.422270 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:37:38.422270 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:38.422270 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:37:38.422270 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:37:38.422270 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:38.422270 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:37:38.422270 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:37:38.422270 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:38.422270 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:38.422270 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:37:38.422270 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:38.423012 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:37:38.423012 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:37:38.423012 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:38.423012 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:38.423012 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:38.423012 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:37:38.423012 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:38.423012 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:38.423012 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:38.423012 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:37:38.423012 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:38.423012 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:37:38.423012 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:37:38.423012 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:38.423012 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:37:38.423012 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:38.423012 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:37:38.423012 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:37:38.423012 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:38.423777 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:37:38.423777 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:38.423777 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:37:42.932776 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:37:42.933877 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:37:42.934295 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:37:42.935943 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:37:42.968282 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:37:42.968710 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:42.969420 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:37:42.969420 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:42.969420 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:37:42.969420 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:37:42.969420 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:42.969420 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:37:42.969420 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:42.969420 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:37:42.970222 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.035 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:42.970222 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:37:42.970222 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:42.970222 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:37:42.970222 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:37:42.970222 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:37:42.970222 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:37:42.970222 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:42.970222 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:42.970222 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:42.970222 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:37:42.970222 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:42.970834 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:42.970834 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:42.970834 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:37:42.970834 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:42.970834 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:37:42.970834 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:37:42.970834 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:42.970834 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:37:42.970834 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:42.970834 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:37:42.970834 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:37:42.970834 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:42.970834 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:37:42.971161 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:42.971161 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:37:42.971161 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:42.971161 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:37:42.971161 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:37:42.971161 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:37:42.971161 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:37:42.971161 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:37:42.971161 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:42.971161 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:37:42.971161 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:37:42.971161 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:42.971161 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:37:42.971161 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:37:42.971161 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:42.971616 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:42.971616 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:37:42.971616 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:42.971616 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:37:42.971616 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:37:42.971616 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:42.971616 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:42.971616 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:42.971616 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:37:42.971616 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:42.971616 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:42.971616 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:42.971616 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:37:42.971616 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:42.971616 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:37:42.971616 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:37:42.971616 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:42.971616 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:37:42.971616 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:42.972089 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:37:42.972089 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:37:42.972089 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:42.972089 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:37:42.972089 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:42.972089 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:37:43.396514 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:37:43.397018 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:37:43.397284 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:37:43.398290 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:37:43.398705 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:43.399747 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:37:43.399747 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:43.399747 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:37:43.399747 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:37:43.399747 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:43.399747 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:37:43.399747 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:43.399747 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:37:43.401201 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:43.401201 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:37:43.401201 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:43.401201 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:37:43.401201 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:37:43.401201 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:37:43.401201 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:37:43.401201 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:43.401201 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:43.401201 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:43.401201 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:37:43.401201 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:43.401773 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:43.401773 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:43.401773 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:37:43.401773 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:43.401773 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:37:43.401773 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:37:43.401773 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:43.401773 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:37:43.401773 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:43.401773 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:37:43.401773 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:37:43.401773 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:43.401773 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:37:43.402321 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:43.402321 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:37:43.402939 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:43.402939 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:37:43.402939 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:37:43.402939 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:37:43.402939 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:37:43.402939 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:37:43.402939 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:43.402939 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:37:43.402939 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:37:43.402939 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:43.402939 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:37:43.402939 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:37:43.402939 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:43.402939 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:43.402939 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:37:43.402939 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:43.404570 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:37:43.404570 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:37:43.404570 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:43.404570 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:43.404570 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:43.404570 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:37:43.404570 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:43.404570 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:43.404570 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:43.404570 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:37:43.404570 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:43.404570 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:37:43.404570 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:37:43.404570 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:43.404570 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:37:43.404570 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:43.404570 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:37:43.404570 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:37:43.404570 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:43.405607 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:37:43.405607 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:43.405607 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:37:43.405607 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:37:43.405607 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:37:43.405607 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:37:43.405607 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:37:43.406017 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:37:43.406102 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:43.406807 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:37:43.406807 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:43.406807 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:37:43.406807 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:37:43.406807 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:43.406807 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:37:43.406807 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:43.406807 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:37:43.407528 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:43.407528 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:37:43.407528 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:37:43.407528 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:37:43.407528 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:43.407528 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:37:43.407528 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:43.407528 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:37:43.407528 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:43.407528 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:37:43.407528 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:43.407528 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:37:43.407528 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:43.407528 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:37:43.407528 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:37:43.407983 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:43.407983 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:37:43.407983 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:43.407983 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:37:43.407983 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:37:43.407983 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:43.407983 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:37:43.407983 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:43.407983 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:37:43.408395 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:43.408395 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:37:43.408395 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:43.408395 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:37:43.408395 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:37:43.408395 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:37:43.408395 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:37:43.408395 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:37:43.408395 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:37:43.408395 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:37:43.408395 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:43.408395 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:43.408735 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:43.408735 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:37:43.408735 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:43.408735 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:43.408735 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:43.408735 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:37:43.408735 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:43.408735 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:37:43.408735 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:37:43.408735 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:43.408735 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:37:43.408735 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:43.408735 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:37:43.408735 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:37:43.409066 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:43.409066 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:37:43.409066 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:43.409066 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:37:43.409496 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:43.409496 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:37:43.409496 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:37:43.409496 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:37:43.409496 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:37:43.409496 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:37:43.409496 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:43.409496 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:37:43.409496 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:37:43.409496 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:43.409496 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:37:43.409496 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:37:43.409496 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:43.409496 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:43.409496 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:37:43.409496 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:43.409977 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:37:43.409977 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:37:43.409977 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:37:43.409977 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:37:43.409977 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:37:43.409977 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:43.409977 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:43.409977 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:43.409977 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:37:43.409977 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:43.409977 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:43.409977 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:43.409977 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:37:43.409977 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:43.409977 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:37:43.409977 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:37:43.409977 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:43.409977 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:37:43.410749 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:43.410749 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:37:43.410749 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:37:43.410749 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:43.410749 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:37:43.410749 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:43.410749 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:37:43.411864 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:37:43.412176 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:37:43.412277 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:37:43.412431 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:37:43.412766 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:37:43.412986 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:43.413447 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:37:43.413447 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:43.413447 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:37:43.413447 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:37:43.413447 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:43.413447 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:37:43.413447 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:43.413447 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:37:43.414185 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:43.414185 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:37:43.414185 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:37:43.414185 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:37:43.414185 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:43.414185 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:37:43.414185 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:43.414185 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:37:43.414185 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:43.414185 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:37:43.414185 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:43.414185 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:37:43.414185 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:43.414185 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:37:43.414185 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:37:43.414593 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:43.414593 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:37:43.414593 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:43.414593 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:37:43.414593 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:37:43.414593 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:43.414593 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:37:43.414593 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:43.414593 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:37:43.415129 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:43.415129 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:37:43.415129 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:43.415129 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:37:43.415129 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:37:43.415129 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:37:43.415129 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:37:43.415129 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:37:43.415129 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:37:43.415129 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:43.415129 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:43.415434 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:43.415434 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:37:43.415434 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:43.415434 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:43.415434 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:43.415434 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:37:43.415434 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:43.415434 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:37:43.415434 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:37:43.415434 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:43.415434 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:37:43.415434 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:43.415434 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:37:43.415434 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:37:43.415778 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:43.415778 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:37:43.415778 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:43.415778 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:37:43.416625 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:43.416625 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:37:43.416625 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:37:43.416625 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:37:43.416625 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:37:43.416625 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:37:43.416625 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:43.416625 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:37:43.416625 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:37:43.416625 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:43.416625 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:37:43.416625 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:37:43.416625 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:43.416625 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:43.416625 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:37:43.416625 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:43.416625 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:37:43.417070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:37:43.417070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:37:43.417070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:37:43.417070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:43.417070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:43.417070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:43.417070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:37:43.417070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:43.417070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:43.417070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:43.417070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:37:43.417070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:43.417070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:37:43.417070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:37:43.417070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:43.417070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:37:43.417070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:43.417070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:37:43.417070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:37:43.417531 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:43.417531 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:37:43.417531 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:43.417531 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:37:43.418947 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:37:43.419171 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:37:43.419304 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:37:43.419489 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:37:43.419757 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:37:43.420099 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:37:43.420250 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:43.420844 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:37:43.420844 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:43.420844 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:37:43.420844 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:37:43.420844 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:43.420844 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:37:43.420844 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:43.420844 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:37:43.421792 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:43.421792 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:37:43.421792 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:43.421792 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:37:43.421792 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:37:43.421792 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:37:43.421792 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:37:43.421792 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:43.421792 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:43.421792 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:43.421792 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:37:43.421792 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:43.422307 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:43.422307 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:43.422307 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:37:43.422307 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:43.422307 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:37:43.422307 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:37:43.422307 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:43.422307 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:37:43.422307 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:43.422307 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:37:43.422307 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:37:43.422307 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:43.422307 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:37:43.422798 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:43.422798 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:37:43.422798 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:43.422798 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:37:43.422798 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:37:43.422798 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:37:43.422798 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:37:43.422798 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:37:43.422798 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:43.422798 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:37:43.422798 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:37:43.422798 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:43.422798 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:37:43.422798 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:37:43.422798 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:43.423163 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:43.423163 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:37:43.423163 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:43.423163 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:37:43.423163 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:37:43.423163 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:43.423163 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:43.423163 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:43.423163 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:37:43.423163 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:43.423163 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:43.423163 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:43.423163 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:37:43.423163 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:43.423163 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:37:43.423163 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:37:43.423163 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:43.423163 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:37:43.423163 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:43.423628 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:37:43.423628 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:37:43.423628 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:43.423628 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:37:43.423628 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:43.423628 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:37:47.933581 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:37:47.934414 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:37:47.934480 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:37:47.935932 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:37:47.938125 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG dbcounter [-] [106123] Writing DB stats neutron:SELECT=240 {{(pid=106123) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} May 18 18:37:47.965945 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:37:47.966306 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:47.967179 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:37:47.967179 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:47.967179 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:37:47.967179 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:37:47.967179 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:47.967179 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:37:47.967179 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:47.967179 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:37:47.967903 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.033 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:47.967903 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:37:47.967903 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:47.967903 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:37:47.967903 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:37:47.967903 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:37:47.967903 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:37:47.967903 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:47.967903 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:47.967903 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:47.967903 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:37:47.967903 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:47.968819 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:47.968819 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:47.968819 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:37:47.968819 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:47.968819 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:37:47.968819 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:37:47.968819 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:47.968819 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:37:47.968819 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:47.968819 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:37:47.968819 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:37:47.968819 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:47.968819 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:37:47.969372 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:47.969372 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:37:47.969372 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:47.969372 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:37:47.969372 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:37:47.969372 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:37:47.969372 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:37:47.969372 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:37:47.969372 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:47.969372 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:37:47.969372 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:37:47.969372 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:47.969372 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:37:47.969372 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:37:47.969372 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:47.970006 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:47.970006 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:37:47.970006 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:47.970006 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:37:47.970006 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:37:47.970006 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:47.970006 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:47.970006 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:47.970006 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:37:47.970006 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:47.970006 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:47.970006 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:47.970006 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:37:47.970006 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:47.970006 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:37:47.970006 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:37:47.970006 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:47.970006 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:37:47.970006 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:47.970785 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:37:47.970785 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:37:47.970785 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:47.970785 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:37:47.970785 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:47.970785 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:37:48.397368 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:37:48.398116 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:37:48.398418 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:37:48.399710 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:37:48.400090 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:48.401188 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:37:48.401188 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:48.401188 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:37:48.401188 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:37:48.401188 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:48.401188 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:37:48.401188 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:48.401188 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:37:48.402988 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:48.402988 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:37:48.402988 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:48.402988 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:37:48.402988 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:37:48.402988 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:37:48.402988 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:37:48.402988 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:48.402988 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:48.402988 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:48.402988 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:37:48.402988 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:48.403565 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:48.403565 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:48.403565 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:37:48.403565 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:48.403565 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:37:48.403565 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:37:48.403565 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:48.403565 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:37:48.403565 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:48.403565 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:37:48.403565 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:37:48.403565 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:48.403565 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:37:48.404091 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:48.404091 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:37:48.405021 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:48.405021 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:37:48.405021 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:37:48.405021 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:37:48.405021 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:37:48.405021 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:37:48.405021 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:48.405021 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:37:48.405021 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:37:48.405021 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:48.405021 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:37:48.405021 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:37:48.405021 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:48.405021 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:48.405021 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:37:48.405021 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:48.405881 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:37:48.405881 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:37:48.405881 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:48.405881 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:48.405881 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:48.405881 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:37:48.405881 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:48.405881 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:48.405881 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:48.405881 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:37:48.405881 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:48.405881 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:37:48.405881 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:37:48.405881 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:48.405881 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:37:48.405881 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:48.405881 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:37:48.405881 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:37:48.405881 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:48.406658 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:37:48.406658 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:48.406658 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:37:48.406658 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:37:48.406658 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:37:48.406658 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:37:48.406658 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:37:48.406658 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:37:48.407064 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:48.407839 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:37:48.407839 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:48.407839 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:37:48.407839 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:37:48.407839 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:48.407839 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:37:48.407839 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:48.407839 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:37:48.409031 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:48.409031 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:37:48.409031 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:37:48.409031 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:37:48.409031 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:48.409031 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:37:48.409031 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:48.409031 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:37:48.409031 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:48.409031 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:37:48.409031 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:48.409031 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:37:48.409031 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:48.409031 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:37:48.409031 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:37:48.409485 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:48.409485 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:37:48.409485 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:48.409485 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:37:48.409485 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:37:48.409485 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:48.409485 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:37:48.409485 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:48.409485 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:37:48.410600 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:48.410600 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:37:48.410600 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:48.410600 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:37:48.410600 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:37:48.410600 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:37:48.410600 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:37:48.410600 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:37:48.410600 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:37:48.410600 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:37:48.410600 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:48.410600 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:48.410927 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:48.410927 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:37:48.410927 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:48.410927 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:48.410927 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:48.410927 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:37:48.410927 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:48.410927 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:37:48.410927 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:37:48.410927 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:48.410927 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:37:48.410927 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:48.410927 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:37:48.410927 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:37:48.411256 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:48.411256 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:37:48.411256 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:48.411256 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:37:48.412428 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:48.412428 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:37:48.412428 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:37:48.412428 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:37:48.412428 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:37:48.412428 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:37:48.412428 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:48.412428 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:37:48.412428 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:37:48.412428 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:48.412428 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:37:48.412428 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:37:48.412428 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:48.412428 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:48.412428 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:37:48.412428 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:48.412871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:37:48.412871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:37:48.412871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:37:48.412871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:37:48.412871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:37:48.412871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:48.412871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:48.412871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:48.412871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:37:48.412871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:48.412871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:48.412871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:48.412871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:37:48.412871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:48.412871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:37:48.412871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:37:48.412871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:48.412871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:37:48.413362 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:48.413362 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:37:48.413362 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:37:48.413362 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:48.413362 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:37:48.413362 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:48.413362 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:37:48.413362 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:37:48.413362 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:37:48.413623 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:37:48.413623 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:37:48.414149 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:37:48.414627 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:48.415530 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:37:48.415530 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:48.415530 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:37:48.415530 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:37:48.415530 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:48.415530 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:37:48.415530 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:48.415530 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:37:48.416302 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:48.416302 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:37:48.416302 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:37:48.416302 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:37:48.416302 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:48.416302 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:37:48.416302 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:48.416302 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:37:48.416302 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:48.416302 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:37:48.416302 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:48.416302 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:37:48.416302 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:48.416302 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:37:48.416302 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:37:48.416755 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:48.416755 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:37:48.416755 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:48.416755 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:37:48.416755 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:37:48.416755 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:48.416755 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:37:48.416755 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:48.416755 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:37:48.417215 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:48.417215 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:37:48.417215 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:48.417215 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:37:48.417215 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:37:48.417215 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:37:48.417215 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:37:48.417215 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:37:48.417215 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:37:48.417215 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:48.417215 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:48.417548 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:48.417548 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:37:48.417548 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:48.417548 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:48.417548 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:48.417548 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:37:48.417548 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:48.417548 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:37:48.417548 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:37:48.417548 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:48.417548 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:37:48.417548 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:48.417548 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:37:48.417548 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:37:48.417924 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:48.417924 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:37:48.417924 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:48.417924 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:37:48.418271 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:48.418271 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:37:48.418271 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:37:48.418271 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:37:48.418271 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:37:48.418271 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:37:48.418271 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:48.418271 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:37:48.418271 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:37:48.418271 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:48.418271 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:37:48.418271 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:37:48.418271 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:48.418271 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:48.418271 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:37:48.418271 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:48.418271 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:37:48.418771 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:37:48.418771 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:37:48.418771 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:37:48.418771 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:48.418771 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:48.418771 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:48.418771 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:37:48.418771 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:48.418771 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:48.418771 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:48.418771 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:37:48.418771 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:48.418771 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:37:48.418771 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:37:48.418771 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:48.418771 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:37:48.418771 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:48.418771 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:37:48.418771 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:37:48.419247 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:48.419247 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:37:48.419247 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:48.419247 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:37:48.419376 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:37:48.419646 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:37:48.419757 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:37:48.419922 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:37:48.420426 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:37:48.420851 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:37:48.421065 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:48.421931 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:37:48.421931 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:48.421931 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:37:48.421931 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:37:48.421931 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:48.421931 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:37:48.421931 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:48.421931 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:37:48.422851 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:48.422851 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:37:48.422851 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:48.422851 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:37:48.422851 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:37:48.422851 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:37:48.422851 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:37:48.422851 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:48.422851 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:48.422851 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:48.422851 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:37:48.422851 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:48.423198 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:48.423198 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:48.423198 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:37:48.423198 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:48.423198 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:37:48.423198 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:37:48.423198 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:48.423198 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:37:48.423198 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:48.423198 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:37:48.423198 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:37:48.423198 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:48.423198 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:37:48.423584 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:48.423584 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:37:48.423837 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:48.423837 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:37:48.423837 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:37:48.423837 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:37:48.423837 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:37:48.423837 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:37:48.423837 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:48.423837 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:37:48.423837 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:37:48.423837 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:48.423837 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:37:48.423837 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:37:48.423837 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:48.423837 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:48.423837 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:37:48.423837 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:48.424516 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:37:48.424516 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:37:48.424516 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:48.424516 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:48.424516 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:48.424516 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:37:48.424516 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:48.424516 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:48.424516 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:48.424516 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:37:48.424516 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:48.424516 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:37:48.424516 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:37:48.424516 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:48.424516 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:37:48.424516 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:48.424516 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:37:48.424516 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:37:48.424516 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:48.425117 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:37:48.425117 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:48.425117 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:37:52.934335 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:37:52.935219 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:37:52.935467 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:37:52.937071 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:37:52.981020 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:37:52.981347 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:52.982453 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:37:52.982453 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:52.982453 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:37:52.982453 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:37:52.982453 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:52.982453 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:37:52.982453 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:52.982453 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:37:52.983164 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.047 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:52.983164 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:37:52.983164 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:52.983164 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:37:52.983164 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:37:52.983164 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:37:52.983164 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:37:52.983164 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:52.983164 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:52.983164 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:52.983164 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:37:52.983164 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:52.983857 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:52.983857 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:52.983857 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:37:52.983857 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:52.983857 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:37:52.983857 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:37:52.983857 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:52.983857 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:37:52.983857 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:52.983857 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:37:52.983857 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:37:52.983857 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:52.983857 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:37:52.984825 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:52.984825 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:37:52.984825 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:52.984825 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:37:52.984825 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:37:52.984825 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:37:52.984825 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:37:52.984825 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:37:52.984825 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:52.984825 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:37:52.984825 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:37:52.984825 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:52.984825 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:37:52.984825 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:37:52.984825 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:52.985498 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:52.985498 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:37:52.985498 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:52.985498 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:37:52.985498 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:37:52.985498 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:52.985498 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:52.985498 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:52.985498 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:37:52.985498 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:52.985498 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:52.985498 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:52.985498 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:37:52.985498 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:52.985498 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:37:52.985498 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:37:52.985498 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:52.985498 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:37:52.985498 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:52.986310 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:37:52.986310 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:37:52.986310 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:52.986310 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:37:52.986310 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:52.986310 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:37:53.398303 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:37:53.399028 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:37:53.399238 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:37:53.400496 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:37:53.400926 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:53.401933 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:37:53.401933 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:53.401933 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:37:53.401933 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:37:53.401933 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:53.401933 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:37:53.401933 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:53.401933 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:37:53.403032 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:53.403032 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:37:53.403032 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:53.403032 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:37:53.403032 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:37:53.403032 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:37:53.403032 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:37:53.403032 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:53.403032 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:53.403032 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:53.403032 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:37:53.403032 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:53.404230 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:53.404230 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:53.404230 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:37:53.404230 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:53.404230 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:37:53.404230 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:37:53.404230 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:53.404230 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:37:53.404230 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:53.404230 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:37:53.404230 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:37:53.404230 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:53.404230 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:37:53.405504 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:53.405504 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:37:53.405504 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:53.405504 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:37:53.405504 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:37:53.405504 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:37:53.405504 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:37:53.405504 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:37:53.405504 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:53.405504 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:37:53.405504 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:37:53.405504 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:53.405504 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:37:53.405504 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:37:53.405504 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:53.406177 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:53.406177 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:37:53.406177 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:53.406177 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:37:53.406177 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:37:53.406177 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:53.406177 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:53.406177 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:53.406177 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:37:53.406177 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:53.406177 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:53.406177 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:53.406177 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:37:53.406177 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:53.406177 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:37:53.406177 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:37:53.406177 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:53.406177 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:37:53.406177 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:53.407010 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:37:53.407010 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:37:53.407010 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:53.407010 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:37:53.407010 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:53.407010 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:37:53.407010 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:37:53.407010 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:37:53.407010 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:37:53.407010 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:37:53.407522 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:37:53.407522 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:53.407777 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:37:53.407777 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:53.407777 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:37:53.407777 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:37:53.407777 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:53.407777 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:37:53.407777 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:53.407777 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:37:53.408613 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:53.408613 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:37:53.408613 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:37:53.408613 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:37:53.408613 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:53.408613 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:37:53.408613 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:53.408613 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:37:53.408613 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:53.408613 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:37:53.408613 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:53.408613 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:37:53.408613 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:53.408613 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:37:53.408613 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:37:53.409312 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:53.409312 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:37:53.409312 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:53.409312 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:37:53.409312 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:37:53.409312 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:53.409312 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:37:53.409312 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:53.409312 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:37:53.409746 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:53.409746 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:37:53.409746 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:53.409746 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:37:53.409746 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:37:53.409746 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:37:53.409746 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:37:53.409746 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:37:53.409746 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:37:53.409746 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:37:53.409746 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:53.409746 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:53.410284 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:53.410284 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:37:53.410284 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:53.410284 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:53.410284 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:53.410284 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:37:53.410284 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:53.410284 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:37:53.410284 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:37:53.410284 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:53.410284 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:37:53.410284 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:53.410284 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:37:53.410284 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:37:53.410898 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:53.410898 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:37:53.410898 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:53.410898 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:37:53.410898 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:53.410898 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:37:53.410898 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:37:53.410898 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:37:53.410898 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:37:53.410898 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:37:53.410898 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:53.410898 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:37:53.410898 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:37:53.410898 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:53.411541 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:37:53.411541 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:37:53.411541 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:53.411541 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:53.411541 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:37:53.411541 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:53.411541 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:37:53.411541 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:37:53.411541 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:37:53.411541 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:37:53.411541 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:37:53.411541 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:53.411541 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:53.411541 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:53.411541 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:37:53.411541 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:53.411541 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:53.411541 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:53.411541 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:37:53.412397 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:53.412397 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:37:53.412397 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:37:53.412397 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:53.412397 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:37:53.412397 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:53.412397 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:37:53.412397 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:37:53.412397 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:53.412397 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:37:53.412397 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:53.412397 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:37:53.413709 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:37:53.414096 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:37:53.414229 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:37:53.414408 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:37:53.414911 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:37:53.415284 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:53.416094 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:37:53.416094 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:53.416094 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:37:53.416094 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:37:53.416094 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:53.416094 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:37:53.416094 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:53.416094 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:37:53.416953 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:53.416953 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:37:53.416953 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:37:53.416953 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:37:53.416953 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:53.416953 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:37:53.416953 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:53.416953 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:37:53.416953 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:53.416953 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:37:53.416953 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:53.416953 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:37:53.416953 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:53.416953 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:37:53.416953 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:37:53.417251 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:53.417251 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:37:53.417251 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:53.417251 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:37:53.417251 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:37:53.417251 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:53.417251 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:37:53.417251 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:53.417251 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:37:53.417937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:53.417937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:37:53.417937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:53.417937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:37:53.417937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:37:53.417937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:37:53.417937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:37:53.417937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:37:53.417937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:37:53.417937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:53.417937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:53.418168 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:53.418168 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:37:53.418168 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:53.418168 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:53.418168 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:53.418168 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:37:53.418168 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:53.418168 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:37:53.418168 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:37:53.418168 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:53.418168 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:37:53.418168 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:53.418168 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:37:53.418168 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:37:53.418449 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:53.418449 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:37:53.418449 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:53.418449 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:37:53.419018 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:53.419018 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:37:53.419018 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:37:53.419018 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:37:53.419018 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:37:53.419018 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:37:53.419018 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:53.419018 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:37:53.419018 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:37:53.419018 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:53.419018 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:37:53.419018 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:37:53.419018 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:53.419018 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:53.419018 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:37:53.419018 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:53.419018 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:37:53.419339 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:37:53.419339 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:37:53.419339 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:37:53.419339 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:53.419339 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:53.419339 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:53.419339 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:37:53.419339 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:53.419339 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:53.419339 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:53.419339 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:37:53.419339 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:53.419339 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:37:53.419339 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:37:53.419339 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:53.419339 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:37:53.419339 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:53.419339 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:37:53.419339 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:37:53.419699 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:53.419699 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:37:53.419699 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:53.419699 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:37:53.419699 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:37:53.420009 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:37:53.420124 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:37:53.420282 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:37:53.420817 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:37:53.421177 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:37:53.421426 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:53.422275 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:37:53.422275 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:53.422275 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:37:53.422275 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:37:53.422275 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:53.422275 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:37:53.422275 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:53.422275 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:37:53.423206 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:53.423206 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:37:53.423206 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:53.423206 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:37:53.423206 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:37:53.423206 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:37:53.423206 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:37:53.423206 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:53.423206 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:53.423206 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:53.423206 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:37:53.423206 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:53.423448 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:53.423448 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:53.423448 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:37:53.423448 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:53.423448 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:37:53.423448 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:37:53.423448 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:53.423448 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:37:53.423448 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:53.423448 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:37:53.423448 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:37:53.423448 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:53.423448 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:37:53.423688 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:53.423688 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:37:53.424289 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:53.424289 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:37:53.424289 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:37:53.424289 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:37:53.424289 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:37:53.424289 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:37:53.424289 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:53.424289 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:37:53.424289 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:37:53.424289 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:53.424289 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:37:53.424289 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:37:53.424289 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:53.424289 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:53.424289 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:37:53.424289 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:53.424611 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:37:53.424611 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:37:53.424611 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:53.424611 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:53.424611 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:53.424611 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:37:53.424611 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:53.424611 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:53.424611 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:53.424611 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:37:53.424611 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:53.424611 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:37:53.424611 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:37:53.424611 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:53.424611 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:37:53.424611 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:53.424611 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:37:53.424611 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:37:53.424611 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:53.424973 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:37:53.424973 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:53.424973 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:37:57.935384 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:37:57.936924 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:37:57.936924 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:37:57.936924 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:37:57.971283 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:37:57.971873 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:57.972945 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:37:57.972945 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:57.972945 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:37:57.972945 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:37:57.972945 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:57.972945 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:37:57.972945 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:57.972945 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:37:57.973795 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.037 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:57.973795 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:37:57.973795 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:57.973795 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:37:57.973795 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:37:57.973795 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:37:57.973795 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:37:57.973795 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:57.973795 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:57.973795 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:57.973795 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:37:57.973795 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:57.974579 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:57.974579 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:57.974579 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:37:57.974579 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:57.974579 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:37:57.974579 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:37:57.974579 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:57.974579 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:37:57.974579 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:57.974579 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:37:57.974579 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:37:57.974579 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:57.974579 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:37:57.975156 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:57.975156 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:37:57.975156 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:57.975156 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:37:57.975156 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:37:57.975156 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:37:57.975156 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:37:57.975156 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:37:57.975156 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:57.975156 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:37:57.975156 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:37:57.975156 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:57.975156 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:37:57.975156 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:37:57.975156 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:57.976039 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:57.976039 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:37:57.976039 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:57.976039 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:37:57.976039 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:37:57.976039 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:57.976039 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:57.976039 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:57.976039 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:37:57.976039 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:57.976039 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:57.976039 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:57.976039 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:37:57.976039 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:57.976039 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:37:57.976039 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:37:57.976039 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:57.976039 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:37:57.976039 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:57.977069 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:37:57.977069 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:37:57.977069 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:57.977069 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:37:57.977069 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:57.977069 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:37:58.399037 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:37:58.399798 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:37:58.400042 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:37:58.401009 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:37:58.402354 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:58.402585 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:37:58.402585 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:58.402585 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:37:58.402585 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:37:58.402585 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:58.402585 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:37:58.402585 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:58.402585 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:37:58.404075 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:58.404075 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:37:58.404075 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:58.404075 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:37:58.404075 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:37:58.404075 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:37:58.404075 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:37:58.404075 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:58.404075 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:58.404075 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:58.404075 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:37:58.404075 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:58.404706 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:58.404706 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:58.404706 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:37:58.404706 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:58.404706 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:37:58.404706 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:37:58.404706 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:58.404706 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:37:58.404706 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:58.404706 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:37:58.404706 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:37:58.404706 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:58.404706 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:37:58.405238 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:58.405238 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:37:58.405854 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:58.405854 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:37:58.405854 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:37:58.405854 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:37:58.405854 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:37:58.405854 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:37:58.405854 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:58.405854 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:37:58.405854 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:37:58.405854 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:58.405854 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:37:58.405854 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:37:58.405854 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:58.405854 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:58.405854 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:37:58.405854 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:58.406465 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:37:58.406465 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:37:58.406465 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:58.406465 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:58.406465 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:58.406465 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:37:58.406465 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:58.406465 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:58.406465 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:58.406465 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:37:58.406465 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:58.406465 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:37:58.406465 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:37:58.406465 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:58.406465 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:37:58.406465 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:58.406465 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:37:58.406465 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:37:58.406465 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:58.406995 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:37:58.406995 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:58.406995 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:37:58.406995 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:37:58.406995 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:37:58.407148 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:37:58.407280 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:37:58.408148 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:37:58.408148 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:58.408343 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:37:58.408343 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:58.408343 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:37:58.408343 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:37:58.408343 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:58.408343 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:37:58.408343 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:58.408343 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:37:58.409060 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:58.409060 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:37:58.409060 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:37:58.409060 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:37:58.409060 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:58.409060 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:37:58.409060 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:58.409060 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:37:58.409060 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:58.409060 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:37:58.409060 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:58.409060 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:37:58.409060 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:58.409060 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:37:58.409060 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:37:58.409516 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:58.409516 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:37:58.409516 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:58.409516 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:37:58.409516 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:37:58.409516 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:58.409516 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:37:58.409516 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:58.409516 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:37:58.409869 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:58.409869 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:37:58.409869 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:58.409869 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:37:58.409869 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:37:58.409869 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:37:58.409869 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:37:58.409869 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:37:58.409869 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:37:58.409869 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:37:58.409869 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:58.409869 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:58.410183 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:58.410183 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:37:58.410183 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:58.410183 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:58.410183 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:58.410183 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:37:58.410183 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:58.410183 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:37:58.410183 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:37:58.410183 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:58.410183 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:37:58.410183 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:58.410183 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:37:58.410183 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:37:58.410618 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:58.410618 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:37:58.410618 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:58.410618 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:37:58.411020 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:58.411020 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:37:58.411020 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:37:58.411020 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:37:58.411020 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:37:58.411020 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:37:58.411020 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:58.411020 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:37:58.411020 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:37:58.411020 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:58.411020 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:37:58.411020 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:37:58.411020 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:58.411020 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:58.411020 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:37:58.411020 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:58.411427 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:37:58.411427 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:37:58.411427 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:37:58.411427 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:37:58.411427 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:37:58.411427 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:58.411427 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:58.411427 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:58.411427 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:37:58.411427 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:58.411427 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:58.411427 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:58.411427 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:37:58.411427 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:58.411427 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:37:58.411427 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:37:58.411427 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:58.411427 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:37:58.411961 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:58.411961 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:37:58.411961 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:37:58.411961 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:58.411961 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:37:58.411961 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:58.411961 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:37:58.414267 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:37:58.414615 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:37:58.414727 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:37:58.414855 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:37:58.415453 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:37:58.415453 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:58.415793 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:37:58.415793 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:58.415793 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:37:58.415793 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:37:58.415793 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:58.415793 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:37:58.415793 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:58.415793 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:37:58.416558 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:58.416558 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:37:58.416558 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:37:58.416558 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:37:58.416558 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:58.416558 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:37:58.416558 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:58.416558 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:37:58.416558 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:58.416558 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:37:58.416558 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:58.416558 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:37:58.416558 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:58.416558 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:37:58.416558 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:37:58.417220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:58.417220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:37:58.417220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:58.417220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:37:58.417220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:37:58.417220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:58.417220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:37:58.417220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:58.417220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:37:58.417654 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:58.417654 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:37:58.417654 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:58.417654 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:37:58.417654 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:37:58.417654 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:37:58.417654 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:37:58.417654 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:37:58.417654 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:37:58.417654 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:58.417654 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:58.418155 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:58.418155 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:37:58.418155 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:58.418155 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:58.418155 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:58.418155 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:37:58.418155 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:58.418155 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:37:58.418155 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:37:58.418155 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:58.418155 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:37:58.418155 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:58.418155 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:37:58.418155 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:37:58.418781 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:58.418781 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:37:58.418781 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:58.418781 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:37:58.418781 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:58.418781 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:37:58.418781 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:37:58.418781 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:37:58.418781 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:37:58.418781 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:37:58.418781 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:58.418781 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:37:58.418781 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:37:58.418781 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:58.419560 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:37:58.419560 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:37:58.419560 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:58.419560 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:58.419560 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:37:58.419560 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:58.419560 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:37:58.419560 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:37:58.419560 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:37:58.419560 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:37:58.419560 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:58.419560 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:58.419560 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:58.419560 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:37:58.419560 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:58.419560 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:58.419560 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:58.419560 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:37:58.419560 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:58.420745 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:37:58.420745 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:37:58.420745 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:58.420745 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:37:58.420745 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:58.420745 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:37:58.420745 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:37:58.420745 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:58.420745 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:37:58.420745 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:58.420745 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:37:58.420745 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:37:58.420745 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:37:58.421275 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:37:58.421275 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:37:58.421275 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:37:58.421395 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:37:58.421503 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:58.422004 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:37:58.422004 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:58.422004 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:37:58.422004 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:37:58.422004 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:58.422004 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:37:58.422004 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:58.422004 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:37:58.422806 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:58.422806 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:37:58.422806 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:58.422806 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:37:58.422806 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:37:58.422806 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:37:58.422806 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:37:58.422806 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:58.422806 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:58.422806 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:58.422806 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:37:58.422806 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:58.423166 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:37:58.423166 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:58.423166 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:37:58.423166 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:58.423166 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:37:58.423166 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:37:58.423166 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:58.423166 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:37:58.423166 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:58.423166 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:37:58.423166 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:37:58.423166 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:58.423166 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:37:58.423532 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:58.423532 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:37:58.423748 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:58.423748 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:37:58.423748 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:37:58.423748 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:37:58.423748 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:37:58.423748 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:37:58.423748 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:58.423748 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:37:58.423748 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:37:58.423748 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:58.423748 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:37:58.423748 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:37:58.423748 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:37:58.423748 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:37:58.423748 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:37:58.423748 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:37:58.424220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:37:58.424220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:37:58.424220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:58.424220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:58.424220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:37:58.424220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:37:58.424220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:37:58.424220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:37:58.424220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:37:58.424220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:37:58.424220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:37:58.424220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:37:58.424220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:37:58.424220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:37:58.424220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:37:58.424220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:37:58.424220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:37:58.424220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:37:58.424220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:37:58.424736 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:37:58.424736 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:37:58.424736 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:38:02.936175 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:38:02.937150 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:38:02.937150 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:38:02.938429 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:38:02.972134 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:38:02.972658 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:02.973684 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:38:02.973684 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:02.973684 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:38:02.973684 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:38:02.973684 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:02.973684 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:38:02.973684 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:02.973684 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:38:02.974822 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.037 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:02.974822 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:38:02.974822 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:02.974822 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:38:02.974822 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:38:02.974822 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:38:02.974822 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:38:02.974822 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:02.974822 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:02.974822 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:02.974822 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:38:02.974822 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:02.975420 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:02.975420 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:02.975420 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:38:02.975420 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:02.975420 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:38:02.975420 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:38:02.975420 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:02.975420 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:38:02.975420 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:02.975420 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:38:02.975420 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:38:02.975420 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:02.975420 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:38:02.976018 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:02.976018 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:38:02.976018 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:02.976018 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:38:02.976018 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:38:02.976018 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:38:02.976018 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:38:02.976018 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:38:02.976018 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:02.976018 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:38:02.976018 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:38:02.976018 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:02.976018 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:38:02.976018 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:38:02.976018 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:02.976778 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:02.976778 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:38:02.976778 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:02.976778 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:38:02.976778 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:38:02.976778 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:02.976778 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:02.976778 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:02.976778 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:38:02.976778 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:02.976778 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:02.976778 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:02.976778 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:38:02.976778 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:02.976778 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:38:02.976778 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:38:02.976778 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:02.976778 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:38:02.976778 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:02.977665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:38:02.977665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:38:02.977665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:02.977665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:38:02.977665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:02.977665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:38:03.399848 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:38:03.400239 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:38:03.400371 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:38:03.401198 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:38:03.401626 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:03.402380 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:38:03.402380 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:03.402380 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:38:03.402380 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:38:03.402380 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:03.402380 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:38:03.402380 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:03.402380 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:38:03.403235 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:03.403235 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:38:03.403235 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:03.403235 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:38:03.403235 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:38:03.403235 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:38:03.403235 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:38:03.403235 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:03.403235 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:03.403235 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:03.403235 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:38:03.403235 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:03.403802 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:03.403802 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:03.403802 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:38:03.403802 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:03.403802 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:38:03.403802 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:38:03.403802 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:03.403802 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:38:03.403802 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:03.403802 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:38:03.403802 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:38:03.403802 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:03.403802 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:38:03.404481 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:03.404481 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:38:03.404481 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:03.404481 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:38:03.404481 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:38:03.404481 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:38:03.404481 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:38:03.404481 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:38:03.404481 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:03.404481 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:38:03.404481 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:38:03.404481 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:03.404481 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:38:03.404481 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:38:03.404481 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:03.405232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:03.405232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:38:03.405232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:03.405232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:38:03.405232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:38:03.405232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:03.405232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:03.405232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:03.405232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:38:03.405232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:03.405232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:03.405232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:03.405232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:38:03.405232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:03.405232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:38:03.405232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:38:03.405232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:03.405232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:38:03.405232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:03.406060 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:38:03.406060 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:38:03.406060 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:03.406060 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:38:03.406060 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:03.406060 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:38:03.406918 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:38:03.407139 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:38:03.407214 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:38:03.407336 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:38:03.407734 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:38:03.408055 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:03.408678 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:38:03.408678 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:03.408678 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:38:03.408678 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:38:03.408678 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:03.408678 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:38:03.408678 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:03.408678 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:38:03.409309 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:03.409309 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:38:03.409309 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:38:03.409309 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:38:03.409309 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:03.409309 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:38:03.409309 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:03.409309 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:38:03.409309 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:03.409309 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:38:03.409309 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:03.409309 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:38:03.409309 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:03.409309 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:38:03.409309 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:38:03.409897 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:03.409897 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:38:03.409897 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:03.409897 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:38:03.409897 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:38:03.409897 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:03.409897 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:38:03.409897 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:03.409897 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:38:03.410158 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:03.410158 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:38:03.410158 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:03.410158 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:38:03.410158 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:38:03.410158 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:38:03.410158 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:38:03.410158 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:38:03.410158 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:38:03.410158 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:38:03.410158 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:03.410158 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:03.410527 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:03.410527 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:38:03.410527 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:03.410527 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:03.410527 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:03.410527 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:38:03.410527 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:03.410527 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:38:03.410527 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:38:03.410527 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:03.410527 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:38:03.410527 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:03.410527 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:38:03.410527 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:38:03.410935 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:03.410935 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:38:03.410935 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:03.410935 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:38:03.410935 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:03.410935 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:38:03.410935 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:38:03.410935 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:38:03.410935 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:38:03.410935 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:38:03.410935 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:03.410935 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:38:03.410935 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:38:03.410935 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:03.411453 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:38:03.411453 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:38:03.411453 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:03.411453 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:03.411453 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:38:03.411453 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:03.411453 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:38:03.411453 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:38:03.411453 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:38:03.411453 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:38:03.411453 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:38:03.411453 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:03.411453 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:03.411453 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:03.411453 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:38:03.411453 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:03.411453 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:03.411453 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:03.411453 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:38:03.412067 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:03.412067 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:38:03.412067 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:38:03.412067 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:03.412067 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:38:03.412067 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:03.412067 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:38:03.412067 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:38:03.412067 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:03.412067 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:38:03.412067 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:03.412067 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:38:03.414696 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:38:03.414946 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:38:03.415111 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:38:03.415111 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:38:03.415546 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:38:03.415857 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:03.416606 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:38:03.416606 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:03.416606 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:38:03.416606 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:38:03.416606 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:03.416606 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:38:03.416606 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:03.416606 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:38:03.417231 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:03.417231 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:38:03.417231 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:38:03.417231 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:38:03.417231 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:03.417231 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:38:03.417231 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:03.417231 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:38:03.417231 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:03.417231 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:38:03.417231 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:03.417231 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:38:03.417231 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:03.417231 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:38:03.417231 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:38:03.417968 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:03.417968 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:38:03.417968 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:03.417968 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:38:03.417968 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:38:03.417968 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:03.417968 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:38:03.417968 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:03.417968 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:38:03.418448 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:03.418448 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:38:03.418448 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:03.418448 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:38:03.418448 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:38:03.418448 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:38:03.418448 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:38:03.418448 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:38:03.418448 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:38:03.418448 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:03.418448 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:03.418981 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:03.418981 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:38:03.418981 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:03.418981 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:03.418981 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:03.418981 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:38:03.418981 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:03.418981 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:38:03.418981 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:38:03.418981 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:03.418981 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:38:03.418981 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:03.418981 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:38:03.418981 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:38:03.419706 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:03.419706 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:38:03.419706 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:03.419706 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:38:03.419706 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:03.419706 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:38:03.419706 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:38:03.419706 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:38:03.419706 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:38:03.419706 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:38:03.419706 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:03.419706 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:38:03.419706 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:38:03.419706 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:03.420484 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:38:03.420484 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:38:03.420484 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:03.420484 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:03.420484 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:38:03.420484 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:03.420484 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:38:03.420484 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:38:03.420484 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:38:03.420484 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:38:03.420484 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:03.420484 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:03.420484 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:03.420484 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:38:03.420484 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:03.420484 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:03.420484 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:03.420484 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:38:03.420484 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:03.421403 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:38:03.421403 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:38:03.421403 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:03.421403 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:38:03.421403 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:03.421403 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:38:03.421403 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:38:03.421403 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:03.421403 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:38:03.421403 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:03.421403 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:38:03.421403 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:38:03.421403 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:38:03.422032 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:38:03.422032 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:38:03.422032 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:38:03.422032 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:38:03.422032 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:03.422494 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:38:03.422494 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:03.422494 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:38:03.422494 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:38:03.422494 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:03.422494 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:38:03.422494 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:03.422494 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:38:03.423198 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:03.423198 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:38:03.423198 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:03.423198 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:38:03.423198 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:38:03.423198 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:38:03.423198 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:38:03.423198 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:03.423198 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:03.423198 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:03.423198 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:38:03.423198 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:03.423693 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:03.423693 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:03.423693 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:38:03.423693 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:03.423693 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:38:03.423693 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:38:03.423693 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:03.423693 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:38:03.423693 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:03.423693 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:38:03.423693 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:38:03.423693 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:03.423693 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:38:03.424141 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:03.424141 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:38:03.424141 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:03.424141 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:38:03.424141 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:38:03.424141 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:38:03.424141 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:38:03.424141 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:38:03.424141 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:03.424141 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:38:03.424141 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:38:03.424141 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:03.424141 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:38:03.424141 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:38:03.424141 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:03.424571 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:03.424571 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:38:03.424571 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:03.424571 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:38:03.424571 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:38:03.424571 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:03.424571 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:03.424571 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:03.424571 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:38:03.424571 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:03.424571 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:03.424571 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:03.424571 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:38:03.424571 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:03.424571 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:38:03.424571 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:38:03.424571 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:03.424571 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:38:03.424571 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:03.425090 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:38:03.425090 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:38:03.425090 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:03.425090 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:38:03.425090 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:03.425090 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:38:07.937014 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:38:07.937891 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:38:07.937891 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:38:07.939273 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:38:07.971045 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:38:07.971694 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:07.972832 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:38:07.972832 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:07.972832 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:38:07.972832 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:38:07.972832 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:07.972832 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:38:07.972832 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:07.972832 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:38:07.973945 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.035 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:07.973945 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:38:07.973945 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:07.973945 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:38:07.973945 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:38:07.973945 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:38:07.973945 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:38:07.973945 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:07.973945 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:07.973945 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:07.973945 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:38:07.973945 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:07.974509 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:07.974509 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:07.974509 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:38:07.974509 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:07.974509 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:38:07.974509 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:38:07.974509 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:07.974509 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:38:07.974509 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:07.974509 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:38:07.974509 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:38:07.974509 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:07.974509 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:38:07.975098 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:07.975098 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:38:07.975098 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:07.975098 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:38:07.975098 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:38:07.975098 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:38:07.975098 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:38:07.975098 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:38:07.975098 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:07.975098 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:38:07.975098 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:38:07.975098 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:07.975098 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:38:07.975098 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:38:07.975098 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:07.975755 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:07.975755 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:38:07.975755 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:07.975755 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:38:07.975755 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:38:07.975755 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:07.975755 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:07.975755 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:07.975755 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:38:07.975755 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:07.975755 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:07.975755 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:07.975755 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:38:07.975755 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:07.975755 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:38:07.975755 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:38:07.975755 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:07.975755 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:38:07.975755 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:07.976646 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:38:07.976646 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:38:07.976646 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:07.976646 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:38:07.976646 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:07.976646 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:38:08.400768 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:38:08.401934 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:38:08.401934 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:38:08.402480 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:38:08.402879 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:08.403842 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:38:08.403842 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:08.403842 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:38:08.403842 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:38:08.403842 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:08.403842 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:38:08.403842 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:08.403842 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:38:08.404975 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:08.404975 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:38:08.404975 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:08.404975 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:38:08.404975 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:38:08.404975 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:38:08.404975 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:38:08.404975 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:08.404975 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:08.404975 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:08.404975 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:38:08.404975 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:08.405595 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:08.405595 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:08.405595 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:38:08.405595 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:08.405595 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:38:08.405595 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:38:08.405595 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:08.405595 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:38:08.405595 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:08.405595 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:38:08.405595 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:38:08.405595 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:08.405595 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:38:08.406112 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:08.406112 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:38:08.406112 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:08.406112 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:38:08.406112 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:38:08.406112 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:38:08.406112 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:38:08.406112 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:38:08.406112 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:08.406112 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:38:08.406112 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:38:08.406112 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:08.406112 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:38:08.406112 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:38:08.406112 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:08.406783 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:08.406783 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:38:08.406783 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:08.406783 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:38:08.406783 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:38:08.406783 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:08.406783 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:08.406783 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:08.406783 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:38:08.406783 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:08.406783 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:08.406783 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:08.406783 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:38:08.406783 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:08.406783 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:38:08.406783 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:38:08.406783 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:08.406783 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:38:08.406783 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:08.407583 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:38:08.407583 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:38:08.407583 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:08.407583 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:38:08.407583 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:08.407583 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:38:08.407583 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:38:08.407944 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:38:08.407944 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:38:08.407944 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:38:08.408860 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:38:08.408860 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:08.409672 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:38:08.409672 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:08.409672 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:38:08.409672 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:38:08.409672 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:08.409672 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:38:08.409672 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:08.409672 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:38:08.410517 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:08.410517 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:38:08.410517 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:38:08.410517 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:38:08.410517 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:08.410517 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:38:08.410517 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:08.410517 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:38:08.410517 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:08.410517 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:38:08.410517 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:08.410517 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:38:08.410517 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:08.410517 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:38:08.410517 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:38:08.411562 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:08.411562 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:38:08.411562 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:08.411562 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:38:08.411562 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:38:08.411562 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:08.411562 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:38:08.411562 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:08.411562 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:38:08.411562 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:08.411562 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:38:08.411562 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:08.411562 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:38:08.411562 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:38:08.412388 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:38:08.412388 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:38:08.412388 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:38:08.412388 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:38:08.412388 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:38:08.412388 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:08.412388 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:08.412388 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:08.412388 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:38:08.412388 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:08.412388 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:08.412388 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:08.412388 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:38:08.413062 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:08.413062 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:38:08.413062 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:38:08.413062 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:08.413062 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:38:08.413062 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:08.413062 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:38:08.413062 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:38:08.413062 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:08.413062 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:38:08.413062 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:08.413062 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:38:08.413713 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:08.413713 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:38:08.413713 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:38:08.413713 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:38:08.413713 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:38:08.413713 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:38:08.413713 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:08.413713 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:38:08.413713 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:38:08.413713 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:08.413713 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:38:08.413713 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:38:08.413713 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:08.413713 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:08.413713 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:38:08.413713 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:08.414540 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:38:08.414540 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:38:08.414540 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:38:08.414540 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:38:08.414540 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:38:08.414540 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:08.414540 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:08.414540 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:08.414540 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:38:08.414540 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:08.414540 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:08.414540 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:08.414540 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:38:08.414540 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:08.414540 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:38:08.414540 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:38:08.414540 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:08.414540 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:38:08.415518 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:08.415518 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:38:08.415518 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:38:08.415518 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:08.415518 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:38:08.415518 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:08.415518 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:38:08.415518 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:38:08.415518 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:38:08.415518 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:38:08.416069 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:38:08.416069 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:38:08.416211 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:08.416828 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:38:08.416828 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:08.416828 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:38:08.416828 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:38:08.416828 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:08.416828 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:38:08.416828 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:08.416828 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:38:08.417669 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:08.417669 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:38:08.417669 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:38:08.417669 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:38:08.417669 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:08.417669 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:38:08.417669 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:08.417669 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:38:08.417669 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:08.417669 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:38:08.417669 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:08.417669 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:38:08.417669 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:08.417669 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:38:08.417669 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:38:08.418024 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:08.418024 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:38:08.418024 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:08.418024 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:38:08.418024 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:38:08.418024 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:08.418024 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:38:08.418024 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:08.418024 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:38:08.418567 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:08.418567 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:38:08.418567 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:08.418567 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:38:08.418567 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:38:08.418567 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:38:08.418567 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:38:08.418567 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:38:08.418567 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:38:08.418567 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:08.418567 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:08.419193 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:08.419193 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:38:08.419193 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:08.419193 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:08.419193 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:08.419193 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:38:08.419193 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:08.419193 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:38:08.419193 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:38:08.419193 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:08.419193 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:38:08.419193 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:08.419193 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:38:08.419193 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:38:08.419917 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:08.419917 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:38:08.419917 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:08.419917 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:38:08.419917 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:08.419917 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:38:08.419917 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:38:08.419917 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:38:08.419917 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:38:08.419917 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:38:08.419917 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:08.419917 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:38:08.419917 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:38:08.419917 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:08.420257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:38:08.420257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:38:08.420257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:08.420257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:08.420257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:38:08.420257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:08.420257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:38:08.420257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:38:08.420257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:38:08.420257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:38:08.420257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:08.420257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:08.420257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:08.420257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:38:08.420257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:08.420257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:08.420257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:08.420257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:38:08.420257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:08.420766 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:38:08.420766 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:38:08.420766 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:08.420766 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:38:08.420766 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:08.420766 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:38:08.420766 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:38:08.420766 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:08.420766 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:38:08.420766 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:08.420766 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:38:08.421071 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:38:08.421496 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:38:08.421496 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:38:08.421706 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:38:08.422074 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:38:08.422453 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:38:08.422635 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:08.423330 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:38:08.423330 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:08.423330 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:38:08.423330 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:38:08.423330 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:08.423330 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:38:08.423330 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:08.423330 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:38:08.424224 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:08.424224 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:38:08.424224 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:08.424224 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:38:08.424224 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:38:08.424224 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:38:08.424224 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:38:08.424224 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:08.424224 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:08.424224 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:08.424224 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:38:08.424224 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:08.424763 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:08.424763 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:08.424763 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:38:08.424763 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:08.424763 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:38:08.424763 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:38:08.424763 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:08.424763 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:38:08.424763 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:08.424763 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:38:08.424763 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:38:08.424763 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:08.424763 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:38:08.425184 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:08.425184 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:38:08.425294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:08.425294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:38:08.425294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:38:08.425294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:38:08.425294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:38:08.425294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:38:08.425294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:08.425294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:38:08.425294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:38:08.425294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:08.425294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:38:08.425294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:38:08.425294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:08.425294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:08.425294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:38:08.425294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:08.425677 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:38:08.425677 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:38:08.425677 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:08.425677 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:08.425677 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:08.425677 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:38:08.425677 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:08.425677 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:08.425677 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:08.425677 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:38:08.425677 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:08.425677 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:38:08.425677 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:38:08.425677 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:08.425677 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:38:08.425677 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:08.425677 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:38:08.425677 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:38:08.425677 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:08.425994 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:38:08.425994 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:08.425994 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:38:12.937856 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:38:12.938788 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:38:12.938788 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:38:12.940000 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:38:12.992862 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:38:12.993267 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:12.994246 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:38:12.994246 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:12.994246 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:38:12.994246 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:38:12.994246 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:12.994246 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:38:12.994246 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:12.994246 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:38:12.994980 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.056 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:12.994980 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:38:12.994980 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:12.994980 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:38:12.994980 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:38:12.994980 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:38:12.994980 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:38:12.994980 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:12.994980 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:12.994980 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:12.994980 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:38:12.994980 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:12.995569 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:12.995569 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:12.995569 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:38:12.995569 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:12.995569 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:38:12.995569 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:38:12.995569 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:12.995569 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:38:12.995569 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:12.995569 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:38:12.995569 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:38:12.995569 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:12.995569 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:38:12.996279 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:12.996279 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:38:12.996279 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:12.996279 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:38:12.996279 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:38:12.996279 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:38:12.996279 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:38:12.996279 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:38:12.996279 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:12.996279 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:38:12.996279 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:38:12.996279 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:12.996279 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:38:12.996279 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:38:12.996279 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:12.997197 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:12.997197 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:38:12.997197 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:12.997197 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:38:12.997197 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:38:12.997197 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:12.997197 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:12.997197 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:12.997197 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:38:12.997197 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:12.997197 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:12.997197 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:12.997197 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:38:12.997197 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:12.997197 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:38:12.997197 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:38:12.997197 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:12.997197 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:38:12.997197 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:12.997936 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:38:12.997936 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:38:12.997936 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:12.997936 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:38:12.997936 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:12.997936 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:38:13.401536 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:38:13.402234 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:38:13.402558 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:38:13.403356 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:38:13.403543 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:13.404136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:38:13.404136 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:13.404136 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:38:13.404136 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:38:13.404136 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:13.404136 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:38:13.404136 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:13.404136 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:38:13.405696 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:13.405696 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:38:13.405696 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:13.405696 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:38:13.405696 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:38:13.405696 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:38:13.405696 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:38:13.405696 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:13.405696 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:13.405696 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:13.405696 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:38:13.405696 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:13.406574 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:13.406574 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:13.406574 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:38:13.406574 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:13.406574 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:38:13.406574 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:38:13.406574 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:13.406574 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:38:13.406574 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:13.406574 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:38:13.406574 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:38:13.406574 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:13.406574 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:38:13.407073 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:13.407073 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:38:13.407073 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:13.407073 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:38:13.407073 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:38:13.407073 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:38:13.407073 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:38:13.407073 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:38:13.407073 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:13.407073 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:38:13.407073 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:38:13.407073 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:13.407073 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:38:13.407073 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:38:13.407073 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:13.407786 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:13.407786 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:38:13.407786 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:13.407786 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:38:13.407786 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:38:13.407786 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:13.407786 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:13.407786 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:13.407786 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:38:13.407786 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:13.407786 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:13.407786 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:13.407786 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:38:13.407786 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:13.407786 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:38:13.407786 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:38:13.407786 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:13.407786 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:38:13.407786 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:13.408797 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:38:13.408797 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:38:13.408797 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:13.408797 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:38:13.408797 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:13.408797 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:38:13.408797 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:38:13.408797 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:38:13.408797 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:38:13.408797 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:38:13.409810 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:38:13.409810 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:13.409810 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:38:13.409810 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:13.409810 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:38:13.409810 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:38:13.409810 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:13.409810 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:38:13.409810 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:13.409810 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:38:13.410705 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:13.410705 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:38:13.410705 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:38:13.410705 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:38:13.410705 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:13.410705 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:38:13.410705 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:13.410705 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:38:13.410705 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:13.410705 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:38:13.410705 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:13.410705 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:38:13.410705 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:13.410705 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:38:13.410705 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:38:13.411660 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:13.411660 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:38:13.411660 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:13.411660 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:38:13.411660 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:38:13.411660 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:13.411660 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:38:13.411660 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:13.411660 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:38:13.411660 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:13.411660 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:38:13.411660 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:13.411660 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:38:13.411660 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:38:13.412225 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:38:13.412225 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:38:13.412225 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:38:13.412225 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:38:13.412225 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:38:13.412225 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:13.412225 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:13.412225 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:13.412225 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:38:13.412225 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:13.412225 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:13.412225 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:13.412225 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:38:13.412791 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:13.412791 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:38:13.412791 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:38:13.412791 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:13.412791 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:38:13.412791 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:13.412791 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:38:13.412791 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:38:13.412791 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:13.412791 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:38:13.412791 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:13.412791 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:38:13.413534 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:13.413534 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:38:13.413534 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:38:13.413534 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:38:13.413534 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:38:13.413534 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:38:13.413534 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:13.413534 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:38:13.413534 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:38:13.413534 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:13.413534 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:38:13.413534 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:38:13.413534 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:13.413534 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:13.413534 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:38:13.413534 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:13.414136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:38:13.414136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:38:13.414136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:38:13.414136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:38:13.414136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:38:13.414136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:13.414136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:13.414136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:13.414136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:38:13.414136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:13.414136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:13.414136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:13.414136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:38:13.414136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:13.414136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:38:13.414136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:38:13.414136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:13.414136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:38:13.414614 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:13.414614 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:38:13.414614 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:38:13.414614 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:13.414614 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:38:13.414614 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:13.414614 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:38:13.415562 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:38:13.415877 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:38:13.415990 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:38:13.416136 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:38:13.416455 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:38:13.416696 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:13.417166 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:38:13.417166 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:13.417166 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:38:13.417166 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:38:13.417166 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:13.417166 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:38:13.417166 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:13.417166 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:38:13.417920 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:13.417920 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:38:13.417920 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:38:13.417920 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:38:13.417920 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:13.417920 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:38:13.417920 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:13.417920 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:38:13.417920 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:13.417920 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:38:13.417920 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:13.417920 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:38:13.417920 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:13.417920 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:38:13.417920 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:38:13.418832 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:13.418832 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:38:13.418832 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:13.418832 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:38:13.418832 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:38:13.418832 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:13.418832 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:38:13.418832 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:13.418832 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:38:13.418832 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:13.418832 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:38:13.418832 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:13.418832 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:38:13.418832 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:38:13.419655 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:38:13.419655 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:38:13.419655 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:38:13.419655 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:38:13.419655 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:13.419655 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:13.419655 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:13.419655 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:38:13.419655 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:13.419655 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:13.419655 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:13.419655 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:38:13.419655 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:13.420391 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:38:13.420391 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:38:13.420391 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:13.420391 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:38:13.420391 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:13.420391 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:38:13.420391 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:38:13.420391 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:13.420391 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:38:13.420391 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:13.420391 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:38:13.420391 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:13.421093 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:38:13.421093 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:38:13.421093 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:38:13.421093 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:38:13.421093 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:38:13.421093 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:13.421093 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:38:13.421093 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:38:13.421093 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:13.421093 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:38:13.421093 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:38:13.421093 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:13.421093 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:13.421093 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:38:13.421093 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:13.421093 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:38:13.421093 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:38:13.421093 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:38:13.421093 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:38:13.421594 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:13.421594 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:13.421594 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:13.421594 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:38:13.421594 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:13.421594 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:13.421594 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:13.421594 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:38:13.421594 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:13.421594 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:38:13.421594 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:38:13.421594 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:13.421594 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:38:13.421594 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:13.421594 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:38:13.421594 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:38:13.421594 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:13.421594 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:38:13.421594 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:13.421594 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:38:13.422091 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:38:13.422091 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:38:13.422091 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:38:13.422091 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:38:13.422413 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:38:13.422701 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:38:13.422701 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:13.423198 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:38:13.423198 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:13.423198 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:38:13.423198 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:38:13.423198 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:13.423198 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:38:13.423198 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:13.423198 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:38:13.424024 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:13.424024 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:38:13.424024 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:13.424024 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:38:13.424024 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:38:13.424024 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:38:13.424024 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:38:13.424024 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:13.424024 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:13.424024 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:13.424024 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:38:13.424024 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:13.424789 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:13.424789 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:13.424789 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:38:13.424789 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:13.424789 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:38:13.424789 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:38:13.424789 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:13.424789 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:38:13.424789 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:13.424789 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:38:13.424789 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:38:13.424789 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:13.424789 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:38:13.425429 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:13.425429 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:38:13.425429 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:13.425429 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:38:13.425429 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:38:13.425429 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:38:13.425429 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:38:13.425429 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:38:13.425429 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:13.425429 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:38:13.425429 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:38:13.425429 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:13.425429 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:38:13.425429 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:38:13.425429 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:13.425828 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:13.425828 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:38:13.425828 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:13.425828 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:38:13.425828 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:38:13.425828 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:13.425828 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:13.425828 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:13.425828 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:38:13.425828 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:13.425828 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:13.425828 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:13.425828 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:38:13.425828 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:13.425828 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:38:13.425828 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:38:13.425828 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:13.425828 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:38:13.425828 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:13.426268 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:38:13.426268 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:38:13.426268 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:13.426268 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:38:13.426268 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:13.426268 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:38:17.938615 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:38:17.940123 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:38:17.940123 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:38:17.941475 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:38:17.982371 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:38:17.982371 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:17.984363 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:38:17.984363 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:17.984363 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:38:17.984363 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:38:17.984363 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:17.984363 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:38:17.984363 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:17.984363 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:38:17.984363 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.043 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:17.984363 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:38:17.984363 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:17.984363 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:38:17.984363 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:38:17.984363 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:38:17.984363 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:38:17.984857 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:17.984857 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:17.984857 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:17.984857 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:38:17.984857 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:17.984857 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:17.984857 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:17.984857 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:38:17.984857 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:17.984857 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:38:17.984857 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:38:17.984857 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:17.984857 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:38:17.984857 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:17.985240 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:38:17.985240 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:38:17.985240 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:17.985240 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:38:17.985240 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:17.985240 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:38:17.986297 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:17.986297 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:38:17.986297 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:38:17.986297 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:38:17.986297 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:38:17.986297 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:38:17.986297 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:17.986297 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:38:17.986297 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:38:17.986297 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:17.986297 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:38:17.986297 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:38:17.986297 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:17.986297 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:17.986297 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:38:17.986297 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:17.986832 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:38:17.986832 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:38:17.986832 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:17.986832 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:17.986832 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:17.986832 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:38:17.986832 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:17.986832 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:17.986832 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:17.986832 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:38:17.986832 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:17.986832 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:38:17.986832 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:38:17.986832 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:17.986832 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:38:17.986832 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:17.986832 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:38:17.986832 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:38:17.986832 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:17.987506 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:38:17.987506 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:17.987506 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:38:18.402823 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:38:18.403723 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:38:18.404019 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:38:18.405351 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:38:18.405720 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:18.406793 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:38:18.406793 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:18.406793 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:38:18.406793 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:38:18.406793 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:18.406793 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:38:18.406793 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:18.406793 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:38:18.408726 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:18.408726 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:38:18.408726 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:18.408726 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:38:18.408726 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:38:18.408726 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:38:18.408726 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:38:18.408726 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:18.408726 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:18.408726 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:18.408726 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:38:18.408726 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:18.409287 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:18.409287 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:18.409287 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:38:18.409287 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:18.409287 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:38:18.409287 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:38:18.409287 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:18.409287 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:38:18.409287 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:18.409287 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:38:18.409287 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:38:18.409287 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:18.409287 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:38:18.409870 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:18.409870 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:38:18.409870 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:38:18.410349 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:18.410349 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:38:18.410349 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:38:18.410349 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:38:18.410349 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:38:18.410349 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:38:18.410349 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:18.410349 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:38:18.410349 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:38:18.410349 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:18.410349 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:38:18.410349 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:38:18.410349 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:18.410349 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:18.410349 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:38:18.410349 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:18.411056 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:38:18.411056 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:38:18.411056 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:18.411056 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:18.411056 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:18.411056 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:38:18.411056 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:18.411056 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:18.411056 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:18.411056 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:38:18.411056 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:18.411056 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:38:18.411056 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:38:18.411056 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:18.411056 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:38:18.411056 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:18.411056 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:38:18.411056 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:38:18.411056 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:18.411965 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:38:18.411965 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:18.411965 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:38:18.411965 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:38:18.411965 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:38:18.411965 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:38:18.411965 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:38:18.411965 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:18.412635 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:38:18.412635 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:18.412635 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:38:18.412635 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:38:18.412635 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:18.412635 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:38:18.412635 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:18.412635 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:38:18.413410 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:18.413410 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:38:18.413410 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:38:18.413410 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:38:18.413410 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:18.413410 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:38:18.413410 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:18.413410 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:38:18.413410 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:18.413410 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:38:18.413410 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:18.413410 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:38:18.413410 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:18.413410 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:38:18.413410 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:38:18.414088 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:18.414088 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:38:18.414088 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:18.414088 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:38:18.414088 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:38:18.414088 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:18.414088 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:38:18.414088 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:18.414088 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:38:18.414529 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:18.414529 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:38:18.414529 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:18.414529 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:38:18.414529 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:38:18.414529 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:38:18.414529 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:38:18.414529 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:38:18.414529 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:38:18.414529 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:38:18.414529 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:18.414529 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:18.415063 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:18.415063 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:38:18.415063 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:18.415063 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:18.415063 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:18.415063 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:38:18.415063 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:18.415063 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:38:18.415063 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:38:18.415063 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:18.415063 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:38:18.415063 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:18.415063 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:38:18.415063 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:38:18.415722 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:18.415722 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:38:18.415722 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:18.415722 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:38:18.415722 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:18.415722 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:38:18.415722 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:38:18.415722 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:38:18.415722 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:38:18.415722 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:38:18.415722 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:18.415722 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:38:18.415722 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:38:18.415722 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:18.416363 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:38:18.416363 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:38:18.416363 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:18.416363 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:18.416363 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:38:18.416363 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:18.416363 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:38:18.416363 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:38:18.416363 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:38:18.416363 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:38:18.416363 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:38:18.416363 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:18.416363 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:18.416363 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:18.416363 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:38:18.416363 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:18.416363 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:18.416363 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:18.416363 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:38:18.417181 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:18.417181 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:38:18.417181 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:38:18.417181 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:18.417181 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:38:18.417181 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:18.417181 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:38:18.417181 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:38:18.417181 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:18.417181 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:38:18.417181 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:18.417181 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:38:18.417181 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:38:18.417181 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:38:18.418255 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:38:18.418255 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:38:18.418255 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:38:18.418255 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:18.418255 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:38:18.418255 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:18.418255 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:38:18.418255 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:38:18.418255 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:18.418255 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:38:18.418255 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:18.418255 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:38:18.419683 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:18.419683 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:38:18.419683 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:38:18.419683 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:38:18.419683 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:18.419683 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:38:18.419683 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:18.419683 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:38:18.419683 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:18.419683 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:38:18.419683 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:18.419683 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:38:18.419683 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:18.419683 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:38:18.419683 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:38:18.420588 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:18.420588 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:38:18.420588 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:18.420588 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:38:18.420588 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:38:18.420588 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:18.420588 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:38:18.420588 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:18.420588 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:38:18.420588 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:18.420588 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:38:18.420588 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:18.420588 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:38:18.420588 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:38:18.421386 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:38:18.421386 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:38:18.421386 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:38:18.421386 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:38:18.421386 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:18.421386 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:18.421386 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:18.421386 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:38:18.421386 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:18.421386 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:18.421386 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:18.421386 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:38:18.421386 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:18.421945 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:38:18.421945 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:38:18.421945 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:18.421945 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:38:18.421945 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:18.421945 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:38:18.421945 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:38:18.421945 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:18.421945 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:38:18.421945 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:18.421945 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:38:18.421945 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:18.422613 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:38:18.422613 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:38:18.422613 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:38:18.422613 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:38:18.422613 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:38:18.422613 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:18.422613 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:38:18.422613 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:38:18.422613 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:18.422613 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:38:18.422613 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:38:18.422613 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:18.422613 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:18.422613 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:38:18.422613 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:18.422613 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:38:18.422613 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:38:18.422613 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:38:18.422613 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:38:18.423371 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:18.423371 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:18.423371 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:18.423371 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:38:18.423371 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:18.423371 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:18.423371 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:18.423371 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:38:18.423371 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:18.423371 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:38:18.423371 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:38:18.423371 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:18.423371 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:38:18.423371 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:18.423371 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:38:18.423371 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:38:18.423371 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:18.423371 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:38:18.423371 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:18.423371 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:38:18.424157 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:38:18.424157 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:38:18.424157 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:38:18.424157 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:38:18.424157 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:38:18.424157 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:38:18.424385 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:18.424385 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:38:18.424385 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:18.424385 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:38:18.424385 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:38:18.424385 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:18.424385 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:38:18.424385 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:18.424385 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:38:18.425896 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:18.425896 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:38:18.425896 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:18.425896 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:38:18.425896 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:38:18.425896 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:38:18.425896 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:38:18.425896 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:18.425896 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:18.425896 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:18.425896 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:38:18.425896 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:18.426426 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:18.426426 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:18.426426 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:38:18.426426 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:18.426426 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:38:18.426426 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:38:18.426426 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:18.426426 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:38:18.426426 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:18.426426 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:38:18.426426 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:38:18.426426 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:18.426426 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:38:18.426946 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:18.426946 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:38:18.427883 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:18.427883 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:38:18.427883 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:38:18.427883 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:38:18.427883 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:38:18.427883 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:38:18.427883 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:18.427883 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:38:18.427883 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:38:18.427883 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:18.427883 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:38:18.427883 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:38:18.427883 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:18.427883 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:18.427883 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:38:18.427883 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:18.428367 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:38:18.428367 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:38:18.428367 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:18.428367 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:18.428367 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:18.428367 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:38:18.428367 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:18.428367 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:18.428367 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:18.428367 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:38:18.428367 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:18.428367 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:38:18.428367 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:38:18.428367 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:18.428367 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:38:18.428367 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:18.428367 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:38:18.428367 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:38:18.428367 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:18.428951 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:38:18.428951 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:18.428951 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:38:22.939409 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:38:22.940189 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:38:22.940189 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:38:22.941087 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:38:22.974507 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:38:22.974846 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:22.975734 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:38:22.975734 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:22.975734 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:38:22.975734 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:38:22.975734 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:22.975734 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:38:22.975734 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:22.975734 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:38:22.976566 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.036 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:22.976566 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:38:22.976566 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:22.976566 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:38:22.976566 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:38:22.976566 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:38:22.976566 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:38:22.976566 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:22.976566 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:22.976566 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:22.976566 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:38:22.976566 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:22.977154 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:22.977154 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:22.977154 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:38:22.977154 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:22.977154 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:38:22.977154 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:38:22.977154 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:22.977154 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:38:22.977154 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:22.977154 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:38:22.977154 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:38:22.977154 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:22.977154 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:38:22.977718 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:22.977718 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:38:22.977718 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:22.977718 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:38:22.977718 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:38:22.977718 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:38:22.977718 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:38:22.977718 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:38:22.977718 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:22.977718 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:38:22.977718 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:38:22.977718 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:22.977718 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:38:22.977718 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:38:22.977718 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:22.978404 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:22.978404 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:38:22.978404 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:22.978404 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:38:22.978404 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:38:22.978404 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:22.978404 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:22.978404 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:22.978404 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:38:22.978404 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:22.978404 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:22.978404 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:22.978404 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:38:22.978404 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:22.978404 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:38:22.978404 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:38:22.978404 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:22.978404 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:38:22.978404 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:22.979188 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:38:22.979188 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:38:22.979188 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:22.979188 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:38:22.979188 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:22.979188 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:38:23.403702 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:38:23.404407 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:38:23.404612 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:38:23.405893 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:38:23.406381 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:23.407410 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:38:23.407410 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:23.407410 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:38:23.407410 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:38:23.407410 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:23.407410 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:38:23.407410 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:23.407410 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:38:23.409070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:23.409070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:38:23.409070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:23.409070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:38:23.409070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:38:23.409070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:38:23.409070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:38:23.409070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:23.409070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:23.409070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:23.409070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:38:23.409070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:23.409824 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:23.409824 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:23.409824 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:38:23.409824 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:23.409824 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:38:23.409824 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:38:23.409824 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:23.409824 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:38:23.409824 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:23.409824 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:38:23.409824 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:38:23.409824 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:23.409824 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:38:23.410428 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:23.410428 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:38:23.410897 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:23.410897 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:38:23.410897 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:38:23.410897 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:38:23.410897 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:38:23.410897 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:38:23.410897 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:23.410897 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:38:23.410897 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:38:23.410897 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:23.410897 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:38:23.410897 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:38:23.410897 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:23.410897 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:23.410897 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:38:23.410897 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:23.411793 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:38:23.411793 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:38:23.411793 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:23.411793 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:23.411793 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:23.411793 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:38:23.411793 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:23.411793 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:23.411793 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:23.411793 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:38:23.411793 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:23.411793 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:38:23.411793 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:38:23.411793 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:23.411793 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:38:23.411793 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:23.411793 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:38:23.411793 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:38:23.411793 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:23.412625 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:38:23.412625 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:23.412625 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:38:23.412625 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:38:23.412625 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:38:23.412625 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:38:23.412625 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:38:23.412625 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:38:23.413005 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:23.413674 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:38:23.413674 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:23.413674 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:38:23.413674 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:38:23.413674 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:23.413674 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:38:23.413674 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:23.413674 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:38:23.414441 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:23.414441 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:38:23.414441 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:38:23.414441 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:38:23.414441 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:23.414441 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:38:23.414441 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:23.414441 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:38:23.414441 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:23.414441 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:38:23.414441 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:23.414441 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:38:23.414441 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:23.414441 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:38:23.414441 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:38:23.415001 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:23.415001 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:38:23.415001 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:23.415001 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:38:23.415001 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:38:23.415001 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:23.415001 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:38:23.415001 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:23.415001 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:38:23.415456 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:23.415456 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:38:23.415456 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:23.415456 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:38:23.415456 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:38:23.415456 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:38:23.415456 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:38:23.415456 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:38:23.415456 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:38:23.415456 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:38:23.415456 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:23.415456 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:23.415957 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:23.415957 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:38:23.415957 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:23.415957 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:23.415957 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:23.415957 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:38:23.415957 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:23.415957 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:38:23.415957 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:38:23.415957 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:23.415957 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:38:23.415957 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:23.415957 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:38:23.415957 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:38:23.416484 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:23.416484 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:38:23.416484 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:23.416484 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:38:23.416642 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:23.416642 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:38:23.416642 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:38:23.416642 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:38:23.416642 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:38:23.416642 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:38:23.416642 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:23.416642 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:38:23.416642 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:38:23.416642 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:23.416642 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:38:23.416642 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:38:23.416642 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:23.416642 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:23.416642 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:38:23.416642 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:23.417259 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:38:23.417259 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:38:23.417259 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:38:23.417259 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:38:23.417259 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:38:23.417259 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:23.417259 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:23.417259 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:23.417259 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:38:23.417259 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:23.417259 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:23.417259 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:23.417259 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:38:23.417259 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:23.417259 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:38:23.417259 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:38:23.417259 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:23.417259 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:38:23.417878 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:23.417878 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:38:23.417878 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:38:23.417878 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:23.417878 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:38:23.417878 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:23.417878 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:38:23.417878 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:38:23.417878 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:38:23.417878 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:38:23.418181 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:38:23.418181 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:38:23.418181 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:23.418657 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:38:23.418657 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:23.418657 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:38:23.418657 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:38:23.418657 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:23.418657 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:38:23.418657 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:23.418657 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:38:23.419484 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:23.419484 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:38:23.419484 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:38:23.419484 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:38:23.419484 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:23.419484 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:38:23.419484 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:23.419484 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:38:23.419484 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:23.419484 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:38:23.419484 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:23.419484 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:38:23.419484 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:23.419484 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:38:23.419484 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:38:23.419924 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:23.419924 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:38:23.419924 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:23.419924 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:38:23.419924 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:38:23.419924 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:23.419924 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:38:23.419924 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:23.419924 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:38:23.420392 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:23.420392 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:38:23.420392 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:23.420392 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:38:23.420392 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:38:23.420392 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:38:23.420392 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:38:23.420392 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:38:23.420392 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:38:23.420392 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:23.420392 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:23.420712 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:23.420712 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:38:23.420712 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:23.420712 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:23.420712 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:23.420712 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:38:23.420712 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:23.420712 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:38:23.420712 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:38:23.420712 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:23.420712 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:38:23.420712 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:23.420712 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:38:23.420712 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:38:23.421111 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:23.421111 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:38:23.421111 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:23.421111 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:38:23.421510 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:23.421510 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:38:23.421510 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:38:23.421510 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:38:23.421510 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:38:23.421510 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:38:23.421510 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:23.421510 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:38:23.421510 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:38:23.421510 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:23.421510 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:38:23.421510 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:38:23.421510 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:23.421510 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:23.421510 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:38:23.421510 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:23.421510 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:38:23.422018 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:38:23.422018 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:38:23.422018 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:38:23.422018 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:23.422018 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:23.422018 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:23.422018 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:38:23.422018 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:23.422018 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:23.422018 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:23.422018 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:38:23.422018 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:23.422018 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:38:23.422018 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:38:23.422018 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:23.422018 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:38:23.422018 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:23.422018 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:38:23.422018 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:38:23.422590 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:23.422590 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:38:23.422590 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:23.422590 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:38:23.422590 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:38:23.422590 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:38:23.422830 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:38:23.422830 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:38:23.423212 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:38:23.423501 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:38:23.423748 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:23.424506 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:38:23.424506 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:23.424506 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:38:23.424506 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:38:23.424506 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:23.424506 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:38:23.424506 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:23.424506 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:38:23.425399 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:23.425399 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:38:23.425399 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:23.425399 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:38:23.425399 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:38:23.425399 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:38:23.425399 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:38:23.425399 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:23.425399 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:23.425399 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:23.425399 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:38:23.425399 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:23.425723 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:23.425723 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:23.425723 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:38:23.425723 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:23.425723 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:38:23.425723 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:38:23.425723 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:23.425723 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:38:23.425723 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:23.425723 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:38:23.425723 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:38:23.425723 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:23.425723 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:38:23.425967 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:23.425967 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:38:23.426402 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:23.426402 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:38:23.426402 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:38:23.426402 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:38:23.426402 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:38:23.426402 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:38:23.426402 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:23.426402 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:38:23.426402 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:38:23.426402 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:23.426402 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:38:23.426402 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:38:23.426402 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:23.426402 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:23.426402 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:38:23.426402 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:23.426749 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:38:23.426749 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:38:23.426749 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:23.426749 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:23.426749 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:23.426749 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:38:23.426749 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:23.426749 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:23.426749 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:23.426749 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:38:23.426749 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:23.426749 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:38:23.426749 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:38:23.426749 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:23.426749 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:38:23.426749 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:23.426749 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:38:23.426749 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:38:23.426749 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:23.427136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:38:23.427136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:23.427136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:38:27.940231 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:38:27.940972 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:38:27.941116 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:38:27.942516 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:38:27.980751 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:38:27.981276 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:27.982505 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:38:27.982505 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:27.982505 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:38:27.982505 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:38:27.982505 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:27.982505 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:38:27.982505 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:27.982505 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:38:27.983920 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.041 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:27.983920 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:38:27.983920 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:27.983920 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:38:27.983920 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:38:27.983920 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:38:27.983920 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:38:27.983920 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:27.983920 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:27.983920 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:27.983920 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:38:27.983920 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:27.984541 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:27.984541 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:27.984541 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:38:27.984541 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:27.984541 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:38:27.984541 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:38:27.984541 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:27.984541 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:38:27.984541 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:27.984541 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:38:27.984541 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:38:27.984541 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:27.984541 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:38:27.985819 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:27.985819 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:38:27.985819 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:27.985819 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:38:27.985819 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:38:27.985819 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:38:27.985819 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:38:27.985819 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:38:27.985819 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:27.985819 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:38:27.985819 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:38:27.985819 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:27.985819 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:38:27.985819 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:38:27.985819 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:27.986806 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:27.986806 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:38:27.986806 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:27.986806 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:38:27.986806 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:38:27.986806 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:27.986806 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:27.986806 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:27.986806 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:38:27.986806 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:27.986806 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:27.986806 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:27.986806 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:38:27.986806 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:27.986806 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:38:27.986806 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:38:27.986806 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:27.986806 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:38:27.986806 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:27.987962 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:38:27.987962 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:38:27.987962 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:27.987962 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:38:27.987962 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:27.987962 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:38:28.404442 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:38:28.405236 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:38:28.405472 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:38:28.406860 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:38:28.407347 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:28.408530 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:38:28.408530 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:28.408530 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:38:28.408530 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:38:28.408530 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:28.408530 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:38:28.408530 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:28.408530 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:38:28.410136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:28.410136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:38:28.410136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:28.410136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:38:28.410136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:38:28.410136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:38:28.410136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:38:28.410136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:28.410136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:28.410136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:28.410136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:38:28.410136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:28.410772 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:28.410772 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:28.410772 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:38:28.410772 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:28.410772 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:38:28.410772 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:38:28.410772 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:28.410772 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:38:28.410772 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:28.410772 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:38:28.410772 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:38:28.410772 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:28.410772 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:38:28.411429 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:28.411429 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:38:28.411832 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:28.411832 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:38:28.411832 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:38:28.411832 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:38:28.411832 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:38:28.411832 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:38:28.411832 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:28.411832 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:38:28.411832 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:38:28.411832 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:28.411832 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:38:28.411832 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:38:28.411832 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:28.411832 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:28.411832 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:38:28.411832 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:28.412751 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:38:28.412751 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:38:28.412751 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:28.412751 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:28.412751 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:28.412751 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:38:28.412751 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:28.412751 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:28.412751 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:28.412751 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:38:28.412751 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:28.412751 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:38:28.412751 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:38:28.412751 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:28.412751 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:38:28.412751 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:28.412751 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:38:28.412751 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:38:28.412751 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:28.413650 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:38:28.413650 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:28.413650 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:38:28.413650 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:38:28.413650 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:38:28.413650 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:38:28.413650 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:38:28.413650 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:38:28.414046 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:28.414046 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:38:28.414046 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:28.414046 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:38:28.414046 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:38:28.414046 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:28.414046 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:38:28.414046 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:28.414046 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:38:28.414875 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:28.414875 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:38:28.414875 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:38:28.414875 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:38:28.414875 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:28.414875 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:38:28.414875 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:28.414875 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:38:28.414875 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:28.414875 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:38:28.414875 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:28.414875 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:38:28.414875 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:28.414875 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:38:28.414875 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:38:28.415644 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:28.415644 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:38:28.415644 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:28.415644 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:38:28.415644 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:38:28.415644 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:28.415644 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:38:28.415644 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:28.415644 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:38:28.415907 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:28.415907 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:38:28.415907 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:28.415907 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:38:28.415907 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:38:28.415907 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:38:28.415907 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:38:28.415907 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:38:28.415907 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:38:28.415907 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:38:28.415907 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:28.415907 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:28.416285 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:28.416285 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:38:28.416285 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:28.416285 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:28.416285 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:28.416285 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:38:28.416285 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:28.416285 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:38:28.416285 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:38:28.416285 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:28.416285 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:38:28.416285 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:28.416285 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:38:28.416285 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:38:28.416717 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:28.416717 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:38:28.416717 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:28.416717 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:38:28.416862 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:28.416862 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:38:28.416862 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:38:28.416862 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:38:28.416862 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:38:28.416862 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:38:28.416862 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:28.416862 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:38:28.416862 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:38:28.416862 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:28.416862 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:38:28.416862 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:38:28.416862 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:28.416862 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:28.416862 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:38:28.416862 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:28.417486 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:38:28.417486 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:38:28.417486 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:38:28.417486 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:38:28.417486 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:38:28.417486 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:28.417486 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:28.417486 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:28.417486 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:38:28.417486 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:28.417486 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:28.417486 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:28.417486 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:38:28.417486 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:28.417486 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:38:28.417486 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:38:28.417486 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:28.417486 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:38:28.418027 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:28.418027 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:38:28.418027 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:38:28.418027 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:28.418027 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:38:28.418027 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:28.418027 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:38:28.418027 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:38:28.418027 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:38:28.418027 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:38:28.418365 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:38:28.418365 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:38:28.418365 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:28.418884 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:38:28.418884 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:28.418884 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:38:28.418884 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:38:28.418884 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:28.418884 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:38:28.418884 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:28.418884 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:38:28.419688 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:28.419688 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:38:28.419688 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:38:28.419688 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:38:28.419688 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:28.419688 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:38:28.419688 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:28.419688 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:38:28.419688 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:28.419688 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:38:28.419688 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:28.419688 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:38:28.419688 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:28.419688 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:38:28.419688 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:38:28.420116 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:28.420116 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:38:28.420116 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:28.420116 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:38:28.420116 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:38:28.420116 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:28.420116 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:38:28.420116 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:28.420116 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:38:28.420555 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:28.420555 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:38:28.420555 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:28.420555 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:38:28.420555 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:38:28.420555 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:38:28.420555 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:38:28.420555 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:38:28.420555 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:38:28.420555 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:28.420555 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:28.420898 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:28.420898 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:38:28.420898 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:28.420898 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:28.420898 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:28.420898 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:38:28.420898 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:28.420898 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:38:28.420898 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:38:28.420898 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:28.420898 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:38:28.420898 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:28.420898 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:38:28.420898 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:38:28.421290 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:28.421290 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:38:28.421290 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:28.421290 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:38:28.421655 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:28.421655 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:38:28.421655 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:38:28.421655 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:38:28.421655 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:38:28.421655 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:38:28.421655 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:28.421655 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:38:28.421655 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:38:28.421655 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:28.421655 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:38:28.421655 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:38:28.421655 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:28.421655 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:28.421655 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:38:28.421655 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:28.421655 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:38:28.422216 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:38:28.422216 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:38:28.422216 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:38:28.422216 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:28.422216 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:28.422216 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:28.422216 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:38:28.422216 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:28.422216 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:28.422216 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:28.422216 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:38:28.422216 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:28.422216 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:38:28.422216 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:38:28.422216 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:28.422216 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:38:28.422216 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:28.422216 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:38:28.422216 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:38:28.422923 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:28.422923 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:38:28.422923 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:28.422923 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:38:28.422923 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:38:28.423095 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:38:28.423095 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:38:28.423338 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:38:28.424001 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:38:28.424290 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:38:28.424922 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:28.425407 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:38:28.425407 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:28.425407 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:38:28.425407 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:38:28.425407 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:28.425407 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:38:28.425407 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:28.425407 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:38:28.426216 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:28.426216 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:38:28.426216 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:28.426216 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:38:28.426216 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:38:28.426216 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:38:28.426216 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:38:28.426216 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:28.426216 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:28.426216 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:28.426216 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:38:28.426216 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:28.426799 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:28.426799 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:28.426799 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:38:28.426799 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:28.426799 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:38:28.426799 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:38:28.426799 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:28.426799 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:38:28.426799 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:28.426799 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:38:28.426799 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:38:28.426799 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:28.426799 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:38:28.427045 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:28.427045 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:38:28.427454 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:28.427454 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:38:28.427454 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:38:28.427454 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:38:28.427454 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:38:28.427454 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:38:28.427454 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:28.427454 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:38:28.427454 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:38:28.427454 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:28.427454 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:38:28.427454 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:38:28.427454 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:28.427454 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:28.427454 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:38:28.427454 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:28.427811 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:38:28.427811 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:38:28.427811 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:28.427811 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:28.427811 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:28.427811 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:38:28.427811 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:28.427811 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:28.427811 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:28.427811 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:38:28.427811 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:28.427811 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:38:28.427811 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:38:28.427811 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:28.427811 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:38:28.427811 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:28.427811 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:38:28.427811 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:38:28.427811 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:28.428175 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:38:28.428175 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:28.428175 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:38:32.941139 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:38:32.942031 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:38:32.942031 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:38:32.944642 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:38:32.987212 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:38:32.987566 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:32.988488 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:38:32.988488 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:32.988488 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:38:32.988488 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:38:32.988488 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:32.988488 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:38:32.988488 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:32.988488 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:38:32.989286 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.046 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:32.989286 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:38:32.989286 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:32.989286 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:38:32.989286 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:38:32.989286 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:38:32.989286 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:38:32.989286 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:32.989286 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:32.989286 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:32.989286 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:38:32.989286 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:32.989700 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:32.989700 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:32.989700 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:38:32.989700 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:32.989700 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:38:32.989700 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:38:32.989700 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:32.989700 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:38:32.989700 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:32.989700 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:38:32.989700 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:38:32.989700 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:32.989700 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:38:32.990096 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:32.990096 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:38:32.990389 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:32.990389 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:38:32.990389 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:38:32.990389 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:38:32.990389 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:38:32.990389 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:38:32.990389 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:32.990389 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:38:32.990389 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:38:32.990389 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:32.990389 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:38:32.990389 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:38:32.990389 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:32.990389 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:32.990389 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:38:32.990389 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:32.990741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:38:32.990741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:38:32.990741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:32.990741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:32.990741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:32.990741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:38:32.990741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:32.990741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:32.990741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:32.990741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:38:32.990741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:32.990741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:38:32.990741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:38:32.990741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:32.990741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:38:32.990741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:32.990741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:38:32.990741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:38:32.990741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:32.991213 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:38:32.991213 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:32.991213 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:38:33.405372 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:38:33.405935 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:38:33.406066 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:38:33.407695 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:38:33.408100 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:33.409154 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:38:33.409154 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:33.409154 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:38:33.409154 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:38:33.409154 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:33.409154 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:38:33.409154 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:33.409154 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:38:33.410265 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:33.410265 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:38:33.410265 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:33.410265 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:38:33.410265 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:38:33.410265 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:38:33.410265 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:38:33.410265 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:33.410265 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:33.410265 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:33.410265 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:38:33.410265 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:33.410546 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:33.410546 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:33.410546 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:38:33.410546 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:33.410546 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:38:33.410546 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:38:33.410546 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:33.410546 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:38:33.410546 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:33.410546 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:38:33.410546 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:38:33.410546 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:33.410546 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:38:33.410791 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:33.410791 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:38:33.411434 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:33.411434 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:38:33.411434 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:38:33.411434 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:38:33.411434 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:38:33.411434 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:38:33.411434 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:33.411434 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:38:33.411434 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:38:33.411434 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:33.411434 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:38:33.411434 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:38:33.411434 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:33.411434 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:33.411434 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:38:33.411434 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:33.411806 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:38:33.411806 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:38:33.411806 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:33.411806 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:33.411806 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:33.411806 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:38:33.411806 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:33.411806 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:33.411806 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:33.411806 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:38:33.411806 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:33.411806 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:38:33.411806 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:38:33.411806 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:33.411806 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:38:33.411806 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:33.411806 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:38:33.411806 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:38:33.411806 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:33.412167 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:38:33.412167 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:33.412167 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:38:33.412784 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:38:33.413257 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:38:33.413489 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:38:33.413693 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:38:33.414298 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:38:33.414754 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:33.415593 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:38:33.415593 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:33.415593 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:38:33.415593 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:38:33.415593 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:33.415593 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:38:33.415593 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:33.415593 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:38:33.417034 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:33.417034 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:38:33.417034 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:38:33.417034 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:38:33.417034 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:33.417034 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:38:33.417034 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:33.417034 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:38:33.417034 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:33.417034 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:38:33.417034 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:33.417034 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:38:33.417034 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:33.417034 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:38:33.417034 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:38:33.417365 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:33.417365 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:38:33.417365 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:33.417365 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:38:33.417365 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:38:33.417365 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:33.417365 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:38:33.417365 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:33.417365 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:38:33.418573 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:33.418573 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:38:33.418573 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:33.418573 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:38:33.418573 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:38:33.418573 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:38:33.418573 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:38:33.418573 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:38:33.418573 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:38:33.418573 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:38:33.418573 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:33.418573 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:33.418837 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:33.418837 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:38:33.418837 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:33.418837 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:33.418837 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:33.418837 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:38:33.418837 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:33.418837 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:38:33.418837 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:38:33.418837 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:33.418837 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:38:33.418837 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:33.418837 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:38:33.418837 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:38:33.419165 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:33.419165 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:38:33.419165 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:33.419165 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:38:33.419165 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:38:33.420636 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:33.420636 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:38:33.420636 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:38:33.420636 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:38:33.420636 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:38:33.420636 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:38:33.420636 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:33.420636 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:38:33.420636 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:38:33.420636 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:33.420636 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:38:33.420636 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:38:33.420636 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:33.420636 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:33.420636 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:38:33.420636 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:33.420949 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:38:33.420949 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:38:33.420949 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:38:33.420949 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:38:33.420949 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:38:33.420949 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:33.420949 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:33.420949 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:33.420949 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:38:33.420949 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:33.420949 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:33.420949 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:33.420949 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:38:33.420949 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:33.420949 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:38:33.420949 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:38:33.420949 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:33.420949 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:38:33.421284 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:33.421284 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:38:33.421284 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:38:33.421284 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:33.421284 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:38:33.421284 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:33.421284 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:38:33.421284 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:38:33.421284 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:38:33.421284 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:38:33.421599 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:38:33.421832 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:33.422403 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:38:33.422403 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:33.422403 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:38:33.422403 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:38:33.422403 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:33.422403 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:38:33.422403 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:33.422403 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:38:33.423219 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:33.423219 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:38:33.423219 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:38:33.423219 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:38:33.423219 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:33.423219 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:38:33.423219 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:33.423219 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:38:33.423219 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:33.423219 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:38:33.423219 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:33.423219 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:38:33.423219 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:33.423219 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:38:33.423219 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:38:33.423493 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:33.423493 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:38:33.423493 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:33.423493 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:38:33.423493 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:38:33.423493 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:33.423493 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:38:33.423493 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:33.423493 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:38:33.424439 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:33.424439 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:38:33.424439 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:33.424439 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:38:33.424439 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:38:33.424439 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:38:33.424439 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:38:33.424439 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:38:33.424439 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:38:33.424439 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:33.424439 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:33.424660 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:33.424660 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:38:33.424660 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:33.424660 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:33.424660 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:33.424660 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:38:33.424660 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:33.424660 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:38:33.424660 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:38:33.424660 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:33.424660 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:38:33.424660 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:33.424660 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:38:33.424660 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:38:33.424905 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:33.424905 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:38:33.424905 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:33.424905 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:38:33.424905 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:38:33.425889 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:33.425889 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:38:33.425889 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:38:33.425889 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:38:33.425889 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:38:33.425889 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:38:33.425889 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:33.425889 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:38:33.425889 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:38:33.425889 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:33.425889 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:38:33.425889 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:38:33.425889 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:33.425889 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:33.425889 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:38:33.425889 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:33.425889 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:38:33.426205 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:38:33.426205 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:38:33.426205 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:38:33.426205 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:33.426205 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:33.426205 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:33.426205 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:38:33.426205 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:33.426205 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:33.426205 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:33.426205 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:38:33.426205 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:33.426205 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:38:33.426205 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:38:33.426205 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:33.426205 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:38:33.426205 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:33.426205 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:38:33.426205 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:38:33.426559 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:33.426559 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:38:33.426559 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:33.426559 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:38:33.426559 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:38:33.426559 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:38:33.426559 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:38:33.427043 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:38:33.427542 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:38:33.427840 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:33.428655 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:38:33.428655 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:33.428655 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:38:33.428655 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:38:33.428655 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:33.428655 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:38:33.428655 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:33.428655 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:38:33.429567 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:33.429567 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:38:33.429567 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:33.429567 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:38:33.429567 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:38:33.429567 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:38:33.429567 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:38:33.429567 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:33.429567 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:33.429567 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:33.429567 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:38:33.429567 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:33.429801 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:33.429801 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:33.429801 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:38:33.429801 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:33.429801 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:38:33.429801 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:38:33.429801 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:33.429801 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:38:33.429801 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:33.429801 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:38:33.429801 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:38:33.429801 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:33.429801 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:38:33.430036 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:33.430036 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:38:33.430708 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:33.430708 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:38:33.430708 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:38:33.430708 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:38:33.430708 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:38:33.430708 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:38:33.430708 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:33.430708 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:38:33.430708 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:38:33.430708 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:33.430708 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:38:33.430708 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:38:33.430708 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:33.430708 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:33.430708 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:38:33.430708 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:33.431017 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:38:33.431017 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:38:33.431017 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:33.431017 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:33.431017 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:33.431017 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:38:33.431017 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:33.431017 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:33.431017 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:33.431017 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:38:33.431017 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:33.431017 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:38:33.431017 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:38:33.431017 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:33.431017 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:38:33.431017 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:33.431017 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:38:33.431017 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:38:33.431017 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:33.431372 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:38:33.431372 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:33.431372 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:38:37.946900 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:38:37.950042 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:38:37.950266 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:38:37.957776 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:38:38.002195 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:38:38.002697 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:38.003837 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:38:38.003837 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:38.003837 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:38:38.003837 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:38:38.003837 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:38.003837 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:38:38.003837 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:38.003837 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:38:38.005457 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.053 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:38.005457 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:38:38.005457 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:38.005457 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:38:38.005457 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:38:38.005457 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:38:38.005457 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:38:38.005457 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:38.005457 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:38.005457 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:38.005457 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:38:38.005457 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:38.005995 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:38.005995 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:38.005995 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:38:38.005995 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:38.005995 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:38:38.005995 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:38:38.005995 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:38.005995 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:38:38.005995 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:38.005995 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:38:38.005995 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:38:38.005995 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:38.005995 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:38:38.006568 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:38.006568 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:38:38.006568 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:38.006568 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:38:38.006568 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:38:38.006568 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:38:38.006568 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:38:38.006568 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:38:38.006568 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:38.006568 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:38:38.006568 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:38:38.006568 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:38.006568 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:38:38.006568 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:38:38.006568 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:38.007719 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:38.007719 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:38:38.007719 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:38.007719 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:38:38.007719 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:38:38.007719 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:38.007719 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:38.007719 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:38.007719 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:38:38.007719 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:38.007719 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:38.007719 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:38.007719 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:38:38.007719 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:38.007719 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:38:38.007719 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:38:38.007719 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:38.007719 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:38:38.007719 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:38.009189 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:38:38.009189 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:38:38.009189 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:38.009189 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:38:38.009189 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:38.009189 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:38:38.416524 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:38:38.416830 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:38:38.418926 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:38:38.419104 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:38:38.419935 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:38:38.420757 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:38.421043 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:38:38.421868 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:38:38.421868 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:38.421868 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:38:38.421868 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:38:38.421868 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:38.421868 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:38:38.421868 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:38.421868 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:38:38.423507 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:38.423507 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:38:38.423507 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:38.423507 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:38:38.423507 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:38:38.423507 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:38:38.423507 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:38:38.423507 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:38.423507 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:38.423507 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:38.423507 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:38:38.423507 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:38.424073 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:38.424073 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:38.424073 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:38:38.424073 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:38.424073 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:38:38.424073 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:38:38.424073 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:38.424073 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:38:38.424073 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:38.424073 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:38:38.424073 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:38:38.424073 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:38.424073 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:38:38.424701 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:38.424701 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:38:38.425243 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:38.425243 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:38:38.425243 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:38:38.425243 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:38:38.425243 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:38:38.425243 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:38:38.425243 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:38.425243 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:38:38.425243 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:38:38.425243 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:38.425243 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:38:38.425243 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:38:38.425243 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:38.425243 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:38.425243 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:38:38.425243 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:38.425682 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:38:38.425682 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:38:38.425682 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:38.425682 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:38.425682 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:38.425682 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:38:38.425682 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:38.425682 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:38.425682 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:38.425682 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:38:38.425682 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:38.425682 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:38:38.425682 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:38:38.425682 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:38.425682 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:38:38.425682 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:38.425682 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:38:38.425682 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:38:38.425682 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:38.426151 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:38:38.426151 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:38.426151 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:38:38.426151 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:38:38.426151 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:38:38.426151 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:38:38.426151 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:38:38.426395 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:38:38.426520 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:38.427049 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:38:38.427049 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:38.427049 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:38:38.427049 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:38:38.427049 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:38.427049 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:38:38.427049 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:38.427049 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:38:38.427807 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:38.427807 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:38:38.427807 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:38:38.427807 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:38:38.427807 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:38.427807 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:38:38.427807 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:38.427807 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:38:38.427807 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:38.427807 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:38:38.427807 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:38.427807 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:38:38.427807 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:38.427807 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:38:38.427807 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:38:38.428216 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:38.428216 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:38:38.428216 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:38.428216 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:38:38.428216 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:38:38.428216 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:38.428216 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:38:38.428216 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:38.428216 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:38:38.428758 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:38.428758 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:38:38.428758 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:38.428758 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:38:38.428758 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:38:38.428758 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:38:38.428758 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:38:38.428758 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:38:38.428758 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:38:38.428758 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:38:38.428758 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:38.428758 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:38.429162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:38.429162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:38:38.429162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:38.429162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:38.429162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:38.429162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:38:38.429162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:38.429162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:38:38.429162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:38:38.429162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:38.429162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:38:38.429162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:38.429162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:38:38.429162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:38:38.429538 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:38.429538 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:38:38.429538 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:38.429538 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:38:38.430844 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:38.430844 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:38:38.430844 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:38:38.430844 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:38:38.430844 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:38:38.430844 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:38:38.430844 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:38.430844 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:38:38.430844 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:38:38.430844 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:38.430844 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:38:38.430844 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:38:38.430844 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:38.430844 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:38.430844 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:38:38.430844 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:38.431301 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:38:38.431301 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:38:38.431301 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:38:38.431301 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:38:38.431301 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:38:38.431301 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:38.431301 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:38.431301 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:38.431301 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:38:38.431301 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:38.431301 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:38.431301 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:38.431301 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:38:38.431301 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:38.431301 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:38:38.431301 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:38:38.431301 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:38.431301 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:38:38.431793 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:38.431793 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:38:38.431793 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:38:38.431793 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:38.431793 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:38:38.431793 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:38.431793 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:38:38.431793 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:38:38.431793 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:38:38.431793 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:38:38.432090 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:38:38.432090 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:38.432090 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:38:38.432090 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:38.432090 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:38:38.432090 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:38:38.432090 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:38.432090 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:38:38.432090 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:38.432090 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:38:38.433412 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:38.433412 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:38:38.433412 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:38:38.433412 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:38:38.433412 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:38.433412 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:38:38.433412 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:38.433412 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:38:38.433412 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:38.433412 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:38:38.433412 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:38.433412 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:38:38.433412 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:38.433412 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:38:38.433412 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:38:38.433845 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:38.433845 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:38:38.433845 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:38.433845 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:38:38.433845 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:38:38.433845 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:38.433845 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:38:38.433845 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:38.433845 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:38:38.434267 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:38.434267 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:38:38.434267 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:38.434267 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:38:38.434267 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:38:38.434267 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:38:38.434267 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:38:38.434267 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:38:38.434267 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:38:38.434267 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:38.434267 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:38.434650 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:38.434650 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:38:38.434650 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:38.434650 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:38.434650 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:38.434650 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:38:38.434650 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:38.434650 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:38:38.434650 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:38:38.434650 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:38.434650 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:38:38.434650 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:38.434650 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:38:38.434650 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:38:38.435033 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:38.435033 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:38:38.435033 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:38.435033 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:38:38.435463 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:38.435463 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:38:38.435463 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:38:38.435463 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:38:38.435463 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:38:38.435463 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:38:38.435463 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:38.435463 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:38:38.435463 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:38:38.435463 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:38.435463 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:38:38.435463 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:38:38.435463 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:38.435463 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:38.435463 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:38:38.435463 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:38.435463 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:38:38.435897 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:38:38.435897 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:38:38.435897 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:38:38.435897 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:38.435897 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:38.435897 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:38.435897 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:38:38.435897 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:38.435897 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:38.435897 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:38.435897 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:38:38.435897 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:38.435897 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:38:38.435897 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:38:38.435897 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:38.435897 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:38:38.435897 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:38.435897 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:38:38.435897 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:38:38.436358 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:38.436358 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:38:38.436358 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:38.436358 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:38:38.436358 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:38:38.436358 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:38:38.436358 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:38:38.436544 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:38:38.436945 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:38:38.437119 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:38.437638 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:38:38.437638 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:38.437638 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:38:38.437638 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:38:38.437638 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:38.437638 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:38:38.437638 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:38.437638 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:38:38.438457 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:38.438457 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:38:38.438457 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:38.438457 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:38:38.438457 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:38:38.438457 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:38:38.438457 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:38:38.438457 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:38.438457 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:38.438457 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:38.438457 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:38:38.438457 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:38.438970 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:38.438970 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:38.438970 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:38:38.438970 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:38.438970 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:38:38.438970 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:38:38.438970 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:38.438970 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:38:38.438970 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:38.438970 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:38:38.438970 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:38:38.438970 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:38.438970 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:38:38.439297 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:38.439297 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:38:38.439428 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:38.439428 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:38:38.439428 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:38:38.439428 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:38:38.439428 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:38:38.439428 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:38:38.439428 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:38.439428 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:38:38.439428 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:38:38.439428 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:38.439428 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:38:38.439428 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:38:38.439428 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:38.439428 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:38.439428 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:38:38.439428 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:38.439877 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:38:38.439877 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:38:38.439877 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:38.439877 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:38.439877 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:38.439877 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:38:38.439877 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:38.439877 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:38.439877 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:38.439877 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:38:38.439877 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:38.439877 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:38:38.439877 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:38:38.439877 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:38.439877 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:38:38.439877 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:38.439877 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:38:38.439877 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:38:38.439877 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:38.440386 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:38:38.440386 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:38.440386 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:38:42.953737 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:38:42.954594 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:38:42.954594 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:38:42.956157 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:38:42.990051 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:38:42.990522 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:42.991543 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:38:42.991543 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:42.991543 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:38:42.991543 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:38:42.991543 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:42.991543 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:38:42.991543 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:42.991543 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:38:42.992573 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.037 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:42.992573 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:38:42.992573 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:42.992573 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:38:42.992573 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:38:42.992573 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:38:42.992573 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:38:42.992573 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:42.992573 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:42.992573 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:42.992573 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:38:42.992573 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:42.993185 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:42.993185 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:42.993185 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:38:42.993185 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:42.993185 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:38:42.993185 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:38:42.993185 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:42.993185 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:38:42.993185 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:42.993185 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:38:42.993185 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:38:42.993185 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:42.993185 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:38:42.993886 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:42.993886 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:38:42.993886 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:42.993886 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:38:42.993886 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:38:42.993886 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:38:42.993886 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:38:42.993886 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:38:42.993886 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:42.993886 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:38:42.993886 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:38:42.993886 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:42.993886 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:38:42.993886 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:38:42.993886 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:42.994609 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:42.994609 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:38:42.994609 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:42.994609 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:38:42.994609 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:38:42.994609 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:42.994609 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:42.994609 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:42.994609 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:38:42.994609 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:42.994609 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:42.994609 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:42.994609 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:38:42.994609 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:42.994609 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:38:42.994609 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:38:42.994609 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:42.994609 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:38:42.994609 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:42.995499 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:38:42.995499 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:38:42.995499 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:42.995499 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:38:42.995499 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:42.995499 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:38:43.418361 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:38:43.418931 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:38:43.419110 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:38:43.420605 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:38:43.421014 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:43.422098 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:38:43.422098 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:43.422098 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:38:43.422098 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:38:43.422098 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:43.422098 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:38:43.422098 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:43.422098 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:38:43.423544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:43.423544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:38:43.423544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:43.423544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:38:43.423544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:38:43.423544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:38:43.423544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:38:43.423544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:43.423544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:43.423544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:43.423544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:38:43.423544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:43.424092 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:43.424092 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:43.424092 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:38:43.424092 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:43.424092 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:38:43.424092 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:38:43.424092 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:43.424092 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:38:43.424092 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:43.424092 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:38:43.424092 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:38:43.424092 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:43.424092 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:38:43.424673 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:43.424673 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:38:43.424673 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:43.424673 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:38:43.424673 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:38:43.424673 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:38:43.424673 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:38:43.424673 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:38:43.424673 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:43.424673 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:38:43.424673 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:38:43.424673 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:43.424673 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:38:43.424673 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:38:43.424673 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:43.425343 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:43.425343 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:38:43.425343 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:43.425343 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:38:43.425343 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:38:43.425343 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:43.425343 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:43.425343 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:43.425343 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:38:43.425343 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:43.425343 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:43.425343 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:43.425343 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:38:43.425343 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:43.425343 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:38:43.425343 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:38:43.425343 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:43.425343 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:38:43.425343 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:43.426171 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:38:43.426171 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:38:43.426171 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:43.426171 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:38:43.426171 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:43.426171 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:38:43.426171 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:38:43.426171 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:38:43.426171 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:38:43.426171 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:38:43.426544 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:38:43.426720 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:43.427375 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:38:43.427375 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:43.427375 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:38:43.427375 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:38:43.427375 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:43.427375 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:38:43.427375 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:43.427375 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:38:43.427927 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:43.427927 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:38:43.427927 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:38:43.427927 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:38:43.427927 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:43.427927 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:38:43.427927 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:43.427927 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:38:43.427927 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:43.427927 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:38:43.427927 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:43.427927 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:38:43.427927 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:43.427927 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:38:43.427927 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:38:43.428371 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:43.428371 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:38:43.428371 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:43.428371 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:38:43.428371 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:38:43.428371 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:43.428371 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:38:43.428371 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:43.428371 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:38:43.428632 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:43.428632 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:38:43.428632 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:43.428632 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:38:43.428632 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:38:43.428632 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:38:43.428632 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:38:43.428632 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:38:43.428632 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:38:43.428632 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:38:43.428632 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:43.428632 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:43.428984 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:43.428984 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:38:43.428984 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:43.428984 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:43.428984 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:43.428984 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:38:43.428984 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:43.428984 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:38:43.428984 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:38:43.428984 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:43.428984 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:38:43.428984 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:43.428984 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:38:43.428984 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:38:43.429367 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:43.429367 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:38:43.429367 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:43.429367 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:38:43.429367 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:43.429367 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:38:43.429367 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:38:43.429367 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:38:43.429367 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:38:43.429367 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:38:43.429367 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:43.429367 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:38:43.429367 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:38:43.429367 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:43.429772 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:38:43.429772 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:38:43.429772 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:43.429772 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:43.429772 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:38:43.429772 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:43.429772 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:38:43.429772 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:38:43.429772 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:38:43.429772 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:38:43.429772 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:38:43.429772 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:43.429772 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:43.429772 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:43.429772 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:38:43.429772 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:43.429772 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:43.429772 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:43.429772 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:38:43.430277 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:43.430277 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:38:43.430277 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:38:43.430277 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:43.430277 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:38:43.430277 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:43.430277 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:38:43.430277 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:38:43.430277 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:43.430277 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:38:43.430277 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:43.430277 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:38:43.430277 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:38:43.430277 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:38:43.430686 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:38:43.430686 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:38:43.430686 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:38:43.430840 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:43.431273 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:38:43.431273 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:43.431273 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:38:43.431273 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:38:43.431273 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:43.431273 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:38:43.431273 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:43.431273 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:38:43.432104 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:43.432104 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:38:43.432104 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:38:43.432104 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:38:43.432104 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:43.432104 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:38:43.432104 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:43.432104 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:38:43.432104 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:43.432104 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:38:43.432104 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:43.432104 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:38:43.432104 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:43.432104 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:38:43.432104 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:38:43.432504 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:43.432504 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:38:43.432504 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:43.432504 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:38:43.432504 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:38:43.432504 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:43.432504 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:38:43.432504 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:43.432504 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:38:43.433027 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:43.433027 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:38:43.433027 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:43.433027 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:38:43.433027 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:38:43.433027 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:38:43.433027 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:38:43.433027 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:38:43.433027 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:38:43.433027 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:43.433027 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:43.433358 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:43.433358 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:38:43.433358 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:43.433358 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:43.433358 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:43.433358 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:38:43.433358 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:43.433358 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:38:43.433358 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:38:43.433358 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:43.433358 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:38:43.433358 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:43.433358 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:38:43.433358 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:38:43.433783 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:43.433783 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:38:43.433783 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:43.433783 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:38:43.434305 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:43.434305 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:38:43.434305 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:38:43.434305 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:38:43.434305 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:38:43.434305 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:38:43.434305 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:43.434305 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:38:43.434305 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:38:43.434305 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:43.434305 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:38:43.434305 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:38:43.434305 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:43.434305 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:43.434305 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:38:43.434305 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:43.434305 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:38:43.434814 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:38:43.434814 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:38:43.434814 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:38:43.434814 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:43.434814 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:43.434814 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:43.434814 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:38:43.434814 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:43.434814 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:43.434814 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:43.434814 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:38:43.434814 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:43.434814 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:38:43.434814 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:38:43.434814 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:43.434814 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:38:43.434814 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:43.434814 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:38:43.434814 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:38:43.435298 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:43.435298 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:38:43.435298 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:43.435298 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:38:43.435691 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:38:43.435880 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:38:43.435983 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:38:43.436131 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:38:43.436453 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:38:43.436665 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:38:43.436781 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:43.437313 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:38:43.437313 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:43.437313 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:38:43.437313 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:38:43.437313 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:43.437313 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:38:43.437313 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:43.437313 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:38:43.438264 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:43.438264 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:38:43.438264 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:43.438264 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:38:43.438264 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:38:43.438264 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:38:43.438264 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:38:43.438264 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:43.438264 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:43.438264 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:43.438264 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:38:43.438264 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:43.438635 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:43.438635 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:43.438635 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:38:43.438635 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:43.438635 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:38:43.438635 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:38:43.438635 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:43.438635 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:38:43.438635 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:43.438635 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:38:43.438635 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:38:43.438635 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:43.438635 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:38:43.438980 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:43.438980 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:38:43.439219 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:43.439219 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:38:43.439219 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:38:43.439219 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:38:43.439219 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:38:43.439219 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:38:43.439219 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:43.439219 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:38:43.439219 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:38:43.439219 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:43.439219 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:38:43.439219 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:38:43.439219 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:43.439219 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:43.439219 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:38:43.439219 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:43.439657 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:38:43.439657 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:38:43.439657 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:43.439657 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:43.439657 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:43.439657 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:38:43.439657 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:43.439657 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:43.439657 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:43.439657 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:38:43.439657 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:43.439657 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:38:43.439657 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:38:43.439657 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:43.439657 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:38:43.439657 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:43.439657 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:38:43.439657 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:38:43.439657 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:43.440153 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:38:43.440153 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:43.440153 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:38:47.954450 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:38:47.955436 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:38:47.955436 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:38:47.956055 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:38:47.957456 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG dbcounter [-] [106123] Writing DB stats neutron:SELECT=240 {{(pid=106123) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} May 18 18:38:47.989416 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:38:47.990175 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:47.991092 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:38:47.991092 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:47.991092 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:38:47.991092 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:38:47.991092 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:47.991092 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:38:47.991092 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:47.991092 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:38:47.992192 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.036 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:47.992192 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:38:47.992192 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:47.992192 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:38:47.992192 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:38:47.992192 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:38:47.992192 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:38:47.992192 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:47.992192 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:47.992192 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:47.992192 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:38:47.992192 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:47.992787 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:47.992787 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:47.992787 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:38:47.992787 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:47.992787 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:38:47.992787 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:38:47.992787 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:47.992787 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:38:47.992787 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:47.992787 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:38:47.992787 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:38:47.992787 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:47.992787 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:38:47.993305 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:47.993305 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:38:47.993305 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:47.993305 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:38:47.993305 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:38:47.993305 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:38:47.993305 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:38:47.993305 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:38:47.993305 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:47.993305 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:38:47.993305 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:38:47.993305 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:47.993305 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:38:47.993305 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:38:47.993305 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:47.993987 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:47.993987 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:38:47.993987 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:47.993987 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:38:47.993987 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:38:47.993987 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:47.993987 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:47.993987 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:47.993987 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:38:47.993987 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:47.993987 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:47.993987 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:47.993987 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:38:47.993987 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:47.993987 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:38:47.993987 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:38:47.993987 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:47.993987 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:38:47.993987 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:47.995250 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:38:47.995250 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:38:47.995250 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:47.995250 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:38:47.995250 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:47.995250 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:38:48.419122 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:38:48.419689 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:38:48.419861 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:38:48.420863 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:38:48.421185 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:48.422036 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:38:48.422036 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:48.422036 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:38:48.422036 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:38:48.422036 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:48.422036 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:38:48.422036 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:48.422036 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:38:48.423100 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:48.423100 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:38:48.423100 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:48.423100 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:38:48.423100 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:38:48.423100 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:38:48.423100 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:38:48.423100 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:48.423100 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:48.423100 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:48.423100 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:38:48.423100 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:48.423916 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:48.423916 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:48.423916 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:38:48.423916 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:48.423916 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:38:48.423916 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:38:48.423916 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:48.423916 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:38:48.423916 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:48.423916 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:38:48.423916 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:38:48.423916 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:48.423916 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:38:48.424593 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:48.424593 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:38:48.424593 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:48.424593 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:38:48.424593 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:38:48.424593 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:38:48.424593 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:38:48.424593 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:38:48.424593 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:48.424593 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:38:48.424593 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:38:48.424593 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:48.424593 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:38:48.424593 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:38:48.424593 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:48.425272 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:48.425272 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:38:48.425272 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:48.425272 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:38:48.425272 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:38:48.425272 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:48.425272 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:48.425272 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:48.425272 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:38:48.425272 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:48.425272 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:48.425272 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:48.425272 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:38:48.425272 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:48.425272 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:38:48.425272 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:38:48.425272 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:48.425272 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:38:48.425272 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:48.426150 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:38:48.426150 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:38:48.426150 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:48.426150 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:38:48.426150 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:48.426150 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:38:48.426150 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:38:48.428068 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:38:48.428068 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:38:48.428068 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:38:48.428068 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:38:48.428068 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:48.428068 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:38:48.428068 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:48.428068 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:38:48.428068 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:38:48.428840 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:48.428840 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:38:48.428840 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:48.428840 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:38:48.429164 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:48.429164 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:38:48.429164 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:38:48.429164 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:38:48.429164 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:48.429164 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:38:48.429164 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:48.429164 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:38:48.429164 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:48.429164 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:38:48.429164 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:48.429164 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:38:48.429164 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:48.429164 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:38:48.429164 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:38:48.430431 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:48.430431 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:38:48.430431 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:48.430431 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:38:48.430431 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:38:48.430431 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:48.430431 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:38:48.430431 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:48.430431 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:38:48.430431 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:48.430431 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:38:48.430431 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:48.430431 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:38:48.430431 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:38:48.431824 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:38:48.431824 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:38:48.431824 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:38:48.431824 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:38:48.431824 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:38:48.431824 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:48.431824 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:48.431824 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:48.431824 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:38:48.431824 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:48.431824 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:48.431824 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:48.431824 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:38:48.432782 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:48.432782 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:38:48.432782 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:38:48.432782 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:48.432782 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:38:48.432782 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:48.432782 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:38:48.432782 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:38:48.432782 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:48.432782 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:38:48.432782 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:48.432782 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:38:48.433231 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:48.433231 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:38:48.433231 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:38:48.433231 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:38:48.433231 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:38:48.433231 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:38:48.433231 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:48.433231 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:38:48.433231 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:38:48.433231 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:48.433231 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:38:48.433231 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:38:48.433231 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:48.433231 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:48.433231 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:38:48.433231 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:48.434762 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:38:48.434762 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:38:48.434762 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:38:48.434762 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:38:48.434762 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:38:48.434762 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:48.434762 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:48.434762 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:48.434762 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:38:48.434762 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:48.434762 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:48.434762 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:48.434762 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:38:48.434762 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:48.434762 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:38:48.434762 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:38:48.434762 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:48.434762 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:38:48.435411 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:48.435411 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:38:48.435411 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:38:48.435411 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:48.435411 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:38:48.435411 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:48.435411 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:38:48.435411 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:38:48.435411 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:38:48.435411 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:38:48.435825 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:38:48.435825 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:38:48.435825 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:48.435825 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:38:48.435825 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:48.435825 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:38:48.435825 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:38:48.435825 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:48.435825 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:38:48.435825 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:48.435825 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:38:48.435825 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:48.436228 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:38:48.436228 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:38:48.436228 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:38:48.436228 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:48.436228 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:38:48.436228 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:48.436228 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:38:48.436228 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:48.436228 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:38:48.436228 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:48.436228 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:38:48.436228 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:48.436228 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:38:48.436228 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:38:48.436228 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:48.436228 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:38:48.436228 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:48.437886 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:38:48.437886 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:38:48.437886 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:48.437886 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:38:48.437886 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:48.437886 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:38:48.437886 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:48.437886 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:38:48.437886 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:48.437886 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:38:48.437886 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:38:48.437886 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:38:48.437886 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:38:48.438232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:38:48.438232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:38:48.438232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:48.438232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:48.438232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:48.438232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:38:48.438232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:48.438232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:48.438232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:48.438232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:38:48.438232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:48.438232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:38:48.438232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:38:48.438570 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:48.438570 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:38:48.438570 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:48.438570 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:38:48.438570 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:38:48.438570 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:48.438570 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:38:48.438570 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:48.438570 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:38:48.438570 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:38:48.440875 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:48.440875 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:38:48.440875 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:38:48.440875 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:38:48.440875 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:38:48.440875 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:38:48.440875 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:48.440875 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:38:48.440875 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:38:48.440875 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:48.440875 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:38:48.440875 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:38:48.440875 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:48.440875 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:48.440875 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:38:48.440875 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:48.440875 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:38:48.441858 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:38:48.441858 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:38:48.441858 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:38:48.441858 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:48.441858 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:48.441858 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:48.441858 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:38:48.441858 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:48.441858 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:48.441858 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:48.441858 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:38:48.441858 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:48.441858 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:38:48.441858 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:38:48.441858 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:48.441858 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:38:48.441858 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:48.441858 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:38:48.441858 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:38:48.442859 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:48.442859 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:38:48.442859 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:48.442859 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:38:48.442859 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:38:48.442859 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:38:48.442859 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:38:48.443482 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:38:48.443919 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:38:48.444129 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:48.445116 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:38:48.445116 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:48.445116 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:38:48.445116 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:38:48.445116 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:48.445116 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:38:48.445116 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:48.445116 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:38:48.446514 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:48.446514 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:38:48.446514 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:48.446514 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:38:48.446514 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:38:48.446514 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:38:48.446514 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:38:48.446514 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:48.446514 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:48.446514 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:48.446514 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:38:48.446514 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:48.446903 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:48.446903 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:48.446903 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:38:48.446903 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:48.446903 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:38:48.446903 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:38:48.446903 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:48.446903 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:38:48.446903 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:48.446903 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:38:48.446903 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:38:48.446903 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:48.446903 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:38:48.447289 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:48.447289 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:38:48.447937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:48.447937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:38:48.447937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:38:48.447937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:38:48.447937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:38:48.447937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:38:48.447937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:48.447937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:38:48.447937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:38:48.447937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:48.447937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:38:48.447937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:38:48.447937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:48.447937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:48.447937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:38:48.447937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:48.448383 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:38:48.448383 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:38:48.448383 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:48.448383 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:48.448383 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:48.448383 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:38:48.448383 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:48.448383 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:48.448383 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:48.448383 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:38:48.448383 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:48.448383 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:38:48.448383 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:38:48.448383 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:48.448383 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:38:48.448383 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:48.448383 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:38:48.448383 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:38:48.448383 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:48.448835 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:38:48.448835 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:48.448835 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:38:52.955402 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:38:52.956222 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:38:52.956488 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:38:52.957766 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:38:52.989566 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:38:52.989947 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:52.990857 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:38:52.990857 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:52.990857 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:38:52.990857 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:38:52.990857 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:52.990857 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:38:52.990857 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:52.990857 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:38:52.991883 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.034 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:52.991883 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:38:52.991883 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:52.991883 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:38:52.991883 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:38:52.991883 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:38:52.991883 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:38:52.991883 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:52.991883 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:52.991883 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:52.991883 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:38:52.991883 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:52.992546 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:52.992546 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:52.992546 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:38:52.992546 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:52.992546 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:38:52.992546 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:38:52.992546 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:52.992546 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:38:52.992546 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:52.992546 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:38:52.992546 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:38:52.992546 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:52.992546 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:38:52.993099 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:52.993099 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:38:52.993099 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:52.993099 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:38:52.993099 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:38:52.993099 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:38:52.993099 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:38:52.993099 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:38:52.993099 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:52.993099 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:38:52.993099 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:38:52.993099 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:52.993099 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:38:52.993099 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:38:52.993099 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:52.993808 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:52.993808 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:38:52.993808 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:52.993808 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:38:52.993808 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:38:52.993808 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:52.993808 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:52.993808 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:52.993808 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:38:52.993808 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:52.993808 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:52.993808 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:52.993808 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:38:52.993808 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:52.993808 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:38:52.993808 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:38:52.993808 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:52.993808 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:38:52.993808 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:52.994677 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:38:52.994677 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:38:52.994677 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:52.994677 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:38:52.994677 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:52.994677 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:38:53.024906 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_for_inconsistencies' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:38:53.025462 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_for_inconsistencies {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:38:53.025626 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Checking Neutron and OVN revision consistency. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:38:53.033155 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Maintenance task: Synchronizing Neutron and OVN databases started {{(pid=106123) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:440}} May 18 18:38:53.033382 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Maintenance task: Number of inconsistencies found at create/update: networks=2, security_groups=1 {{(pid=106123) _log /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:419}} May 18 18:38:53.033474 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Maintenance task: Fixing resource c87833c8-4a0c-4940-8b37-55defa29a810 (type: networks) at create/update {{(pid=106123) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} May 18 18:38:53.070213 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-c87833c8-4a0c-4940-8b37-55defa29a810, resource={'id': 'c87833c8-4a0c-4940-8b37-55defa29a810', 'name': 'public', 'tenant_id': '948b346761bb4236874c86f47a1b6547', 'project_id': '948b346761bb4236874c86f47a1b6547', 'admin_state_up': True, 'mtu': 1430, 'status': 'ACTIVE', 'subnets': ['7cddaefe-671c-42c8-b491-452941ac7074', 'aa9a78c7-2028-4bfd-882a-b6cd5e142951'], 'standard_attr_id': 18, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': True, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'is_default': True, 'tags': [], 'created_at': '2026-05-18T18:33:45Z', 'updated_at': '2026-05-18T18:33:54Z', 'revision_number': 3, 'provider:network_type': 'flat', 'provider:physical_network': 'public', 'provider:segmentation_id': None}, resource_type=networks, if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:38:53.070804 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-c87833c8-4a0c-4940-8b37-55defa29a810, col_values=(('external_ids', {'neutron:network_name': 'public', 'neutron:mtu': '1430', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'flat', 'neutron:provnet-physical-network': 'public'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false', 'fdb_age_threshold': '300', 'broadcast-arps-to-all-routers': 'true'})), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:38:53.071293 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-cfdb2758-3efe-469e-8886-f1bbbc4f5991, external_ids_update=None, columns={'tag': []}, if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:38:53.071699 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:53.072540 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:38:53.072540 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:53.072540 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:38:53.072540 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:38:53.072540 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:53.072540 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:38:53.072540 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:53.072540 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:38:53.073936 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Maintenance task: Failed to fix resource c87833c8-4a0c-4940-8b37-55defa29a810 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:53.073936 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:38:53.073936 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies May 18 18:38:53.073936 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) May 18 18:38:53.073936 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update May 18 18:38:53.073936 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) May 18 18:38:53.073936 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2332, in update_network May 18 18:38:53.073936 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:38:53.073936 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:53.073936 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:53.073936 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:53.074602 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:38:53.074602 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:53.074602 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:53.074602 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:53.074602 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:38:53.074602 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:53.074602 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:38:53.074602 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:38:53.074602 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:53.074602 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:38:53.074602 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:53.074602 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:38:53.074602 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:38:53.074602 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:53.075191 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:38:53.075191 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:53.075191 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:38:53.075191 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Maintenance task: Fixing resource 57ad5e9e-3f36-4459-bc11-2c874585c24d (type: networks) at create/update {{(pid=106123) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} May 18 18:38:53.106407 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-57ad5e9e-3f36-4459-bc11-2c874585c24d, resource={'id': '57ad5e9e-3f36-4459-bc11-2c874585c24d', 'name': 'private', 'tenant_id': '948b346761bb4236874c86f47a1b6547', 'project_id': '948b346761bb4236874c86f47a1b6547', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['05229a28-a9c1-4511-b80e-ae0ea7d0d965', '60919834-7aa5-4e13-9aa4-10a101ac2a78'], 'standard_attr_id': 12, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-05-18T18:33:38Z', 'updated_at': '2026-05-18T18:33:42Z', 'revision_number': 3, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 29742}, resource_type=networks, if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:38:53.106790 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-57ad5e9e-3f36-4459-bc11-2c874585c24d, col_values=(('external_ids', {'neutron:network_name': 'private', 'neutron:mtu': '1372', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:38:53.107362 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-77858cb5-ba15-4032-983c-8dac436dba25, external_ids_update=None, columns={'tag': 29742}, if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:38:53.107599 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:53.108410 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:38:53.108410 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:53.108410 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:38:53.108410 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:38:53.108410 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:53.108410 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:38:53.108410 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:53.108410 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:38:53.109442 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Maintenance task: Failed to fix resource 57ad5e9e-3f36-4459-bc11-2c874585c24d (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:53.109442 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:38:53.109442 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies May 18 18:38:53.109442 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) May 18 18:38:53.109442 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update May 18 18:38:53.109442 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) May 18 18:38:53.109442 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2332, in update_network May 18 18:38:53.109442 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:38:53.109442 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:53.109442 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:53.109442 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:53.109957 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:38:53.109957 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:53.109957 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:53.109957 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:53.109957 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:38:53.109957 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:53.109957 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:38:53.109957 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:38:53.109957 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:53.109957 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:38:53.109957 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:53.109957 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:38:53.109957 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:38:53.109957 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:53.110565 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:38:53.110565 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:53.110565 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:38:53.110565 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Maintenance task: Fixing resource 0f9e7f47-82bf-4126-ac70-00acc7fea232 (type: security_groups) at create/update {{(pid=106123) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} May 18 18:38:53.166656 np0000174525 neutron-ovn-maintenance-worker[106123]: INFO neutron.db.ovn_revision_numbers_db [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Successfully bumped revision number for resource 0f9e7f47-82bf-4126-ac70-00acc7fea232 (type: security_groups) to 9 May 18 18:38:53.166798 np0000174525 neutron-ovn-maintenance-worker[106123]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_for_inconsistencies finished in 0.141 seconds May 18 18:38:53.419848 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:38:53.420356 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:38:53.420657 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:38:53.421807 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:38:53.422162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:53.423154 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:38:53.423154 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:53.423154 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:38:53.423154 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:38:53.423154 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:53.423154 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:38:53.423154 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:53.423154 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:38:53.424977 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:53.424977 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:38:53.424977 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:53.424977 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:38:53.424977 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:38:53.424977 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:38:53.424977 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:38:53.424977 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:53.424977 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:53.424977 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:53.424977 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:38:53.424977 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:53.425649 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:53.425649 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:53.425649 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:38:53.425649 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:53.425649 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:38:53.425649 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:38:53.425649 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:53.425649 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:38:53.425649 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:53.425649 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:38:53.425649 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:38:53.425649 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:53.425649 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:38:53.426170 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:53.426170 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:38:53.426775 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:53.426775 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:38:53.426775 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:38:53.426775 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:38:53.426775 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:38:53.426775 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:38:53.426775 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:53.426775 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:38:53.426775 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:38:53.426775 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:53.426775 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:38:53.426775 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:38:53.426775 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:53.426775 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:53.426775 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:38:53.426775 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:53.427558 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:38:53.427558 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:38:53.427558 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:53.427558 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:53.427558 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:53.427558 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:38:53.427558 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:53.427558 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:53.427558 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:53.427558 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:38:53.427558 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:53.427558 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:38:53.427558 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:38:53.427558 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:53.427558 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:38:53.427558 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:53.427558 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:38:53.427558 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:38:53.427558 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:53.428431 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:38:53.428431 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:53.428431 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:38:53.428431 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:38:53.428431 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:38:53.428431 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:38:53.428431 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:38:53.428818 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:38:53.429058 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:53.429905 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:38:53.429905 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:53.429905 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:38:53.429905 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:38:53.429905 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:53.429905 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:38:53.429905 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:53.429905 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:38:53.431680 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:53.431680 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:38:53.431680 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:38:53.431680 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:38:53.431680 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:53.431680 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:38:53.431680 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:53.431680 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:38:53.431680 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:53.431680 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:38:53.431680 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:53.431680 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:38:53.431680 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:53.431680 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:38:53.431680 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:38:53.432127 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:53.432127 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:38:53.432127 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:53.432127 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:38:53.432127 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:38:53.432127 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:53.432127 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:38:53.432127 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:53.432127 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:38:53.433739 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:53.433739 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:38:53.433739 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:53.433739 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:38:53.433739 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:38:53.433739 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:38:53.433739 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:38:53.433739 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:38:53.433739 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:38:53.433739 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:38:53.433739 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:53.433739 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:53.434098 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:53.434098 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:38:53.434098 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:53.434098 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:53.434098 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:53.434098 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:38:53.434098 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:53.434098 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:38:53.434098 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:38:53.434098 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:53.434098 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:38:53.434098 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:53.434098 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:38:53.434098 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:38:53.434455 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:53.434455 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:38:53.434455 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:53.434455 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:38:53.434455 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:38:53.435162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:53.435162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:38:53.435162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:38:53.435162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:38:53.435162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:38:53.435162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:38:53.435162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:53.435162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:38:53.435162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:38:53.435162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:53.435162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:38:53.435162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:38:53.435162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:53.435162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:53.435162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:38:53.435162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:53.435613 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:38:53.435613 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:38:53.435613 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:38:53.435613 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:38:53.435613 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:38:53.435613 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:53.435613 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:53.435613 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:53.435613 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:38:53.435613 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:53.435613 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:53.435613 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:53.435613 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:38:53.435613 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:53.435613 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:38:53.435613 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:38:53.435613 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:53.435613 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:38:53.436122 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:53.436122 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:38:53.436122 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:38:53.436122 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:53.436122 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:38:53.436122 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:53.436122 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:38:53.436122 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:38:53.436122 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:38:53.436122 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:38:53.436451 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:38:53.436451 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:53.437113 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:38:53.437113 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:53.437113 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:38:53.437113 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:38:53.437113 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:53.437113 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:38:53.437113 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:53.437113 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:38:53.437839 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:53.437839 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:38:53.437839 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:38:53.437839 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:38:53.437839 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:53.437839 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:38:53.437839 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:53.437839 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:38:53.437839 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:53.437839 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:38:53.437839 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:53.437839 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:38:53.437839 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:53.437839 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:38:53.437839 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:38:53.438247 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:53.438247 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:38:53.438247 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:53.438247 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:38:53.438247 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:38:53.438247 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:53.438247 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:38:53.438247 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:53.438247 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:38:53.438710 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:53.438710 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:38:53.438710 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:53.438710 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:38:53.438710 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:38:53.438710 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:38:53.438710 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:38:53.438710 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:38:53.438710 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:38:53.438710 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:53.438710 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:53.439046 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:53.439046 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:38:53.439046 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:53.439046 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:53.439046 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:53.439046 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:38:53.439046 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:53.439046 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:38:53.439046 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:38:53.439046 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:53.439046 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:38:53.439046 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:53.439046 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:38:53.439046 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:38:53.439435 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:53.439435 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:38:53.439435 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:53.439435 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:38:53.439752 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:53.439752 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:38:53.439752 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:38:53.439752 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:38:53.439752 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:38:53.439752 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:38:53.439752 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:53.439752 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:38:53.439752 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:38:53.439752 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:53.439752 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:38:53.439752 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:38:53.439752 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:53.439752 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:53.439752 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:38:53.439752 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:53.439752 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:38:53.440224 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:38:53.440224 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:38:53.440224 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:38:53.440224 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:53.440224 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:53.440224 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:53.440224 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:38:53.440224 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:53.440224 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:53.440224 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:53.440224 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:38:53.440224 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:53.440224 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:38:53.440224 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:38:53.440224 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:53.440224 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:38:53.440224 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:53.440224 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:38:53.440224 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:38:53.440743 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:53.440743 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:38:53.440743 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:53.440743 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:38:53.441340 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:38:53.441659 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:38:53.441766 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:38:53.441910 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:38:53.442222 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:38:53.445555 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:38:53.445671 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:53.446205 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:38:53.446205 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:53.446205 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:38:53.446205 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:38:53.446205 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:53.446205 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:38:53.446205 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:53.446205 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:38:53.447049 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:53.447049 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:38:53.447049 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:53.447049 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:38:53.447049 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:38:53.447049 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:38:53.447049 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:38:53.447049 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:53.447049 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:53.447049 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:53.447049 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:38:53.447049 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:53.447453 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:53.447453 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:53.447453 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:38:53.447453 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:53.447453 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:38:53.447453 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:38:53.447453 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:53.447453 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:38:53.447453 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:53.447453 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:38:53.447453 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:38:53.447453 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:53.447453 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:38:53.447798 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:53.447798 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:38:53.448044 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:53.448044 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:38:53.448044 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:38:53.448044 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:38:53.448044 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:38:53.448044 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:38:53.448044 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:53.448044 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:38:53.448044 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:38:53.448044 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:53.448044 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:38:53.448044 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:38:53.448044 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:53.448044 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:53.448044 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:38:53.448044 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:53.448504 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:38:53.448504 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:38:53.448504 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:53.448504 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:53.448504 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:53.448504 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:38:53.448504 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:53.448504 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:53.448504 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:53.448504 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:38:53.448504 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:53.448504 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:38:53.448504 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:38:53.448504 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:53.448504 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:38:53.448504 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:53.448504 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:38:53.448504 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:38:53.448504 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:53.449033 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:38:53.449033 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:53.449033 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:38:57.956124 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:38:57.956652 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:38:57.956856 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:38:57.958148 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:38:57.988575 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:38:57.988992 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:57.989822 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:38:57.989822 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:57.989822 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:38:57.989822 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:38:57.989822 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:57.989822 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:38:57.989822 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:57.989822 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:38:57.990608 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.033 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:57.990608 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:38:57.990608 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:57.990608 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:38:57.990608 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:38:57.990608 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:38:57.990608 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:38:57.990608 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:57.990608 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:57.990608 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:57.990608 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:38:57.990608 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:57.991210 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:57.991210 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:57.991210 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:38:57.991210 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:57.991210 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:38:57.991210 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:38:57.991210 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:57.991210 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:38:57.991210 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:57.991210 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:38:57.991210 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:38:57.991210 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:57.991210 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:38:57.991836 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:57.991836 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:38:57.991836 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:57.991836 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:38:57.991836 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:38:57.991836 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:38:57.991836 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:38:57.991836 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:38:57.991836 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:57.991836 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:38:57.991836 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:38:57.991836 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:57.991836 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:38:57.991836 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:38:57.991836 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:57.992565 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:57.992565 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:38:57.992565 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:57.992565 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:38:57.992565 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:38:57.992565 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:57.992565 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:57.992565 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:57.992565 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:38:57.992565 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:57.992565 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:57.992565 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:57.992565 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:38:57.992565 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:57.992565 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:38:57.992565 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:38:57.992565 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:57.992565 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:38:57.992565 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:57.993441 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:38:57.993441 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:38:57.993441 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:57.993441 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:38:57.993441 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:57.993441 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:38:58.420631 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:38:58.421256 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:38:58.421511 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:38:58.422659 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:38:58.422998 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:58.423932 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:38:58.423932 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:58.423932 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:38:58.423932 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:38:58.423932 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:58.423932 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:38:58.423932 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:58.423932 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:38:58.425681 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:58.425681 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:38:58.425681 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:58.425681 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:38:58.425681 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:38:58.425681 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:38:58.425681 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:38:58.425681 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:58.425681 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:58.425681 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:58.425681 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:38:58.425681 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:58.426288 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:58.426288 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:58.426288 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:38:58.426288 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:58.426288 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:38:58.426288 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:38:58.426288 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:58.426288 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:38:58.426288 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:58.426288 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:38:58.426288 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:38:58.426288 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:58.426288 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:38:58.426887 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:58.426887 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:38:58.427574 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:58.427574 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:38:58.427574 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:38:58.427574 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:38:58.427574 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:38:58.427574 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:38:58.427574 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:58.427574 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:38:58.427574 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:38:58.427574 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:58.427574 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:38:58.427574 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:38:58.427574 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:58.427574 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:58.427574 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:38:58.427574 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:58.428274 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:38:58.428274 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:38:58.428274 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:58.428274 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:58.428274 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:58.428274 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:38:58.428274 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:58.428274 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:58.428274 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:58.428274 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:38:58.428274 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:58.428274 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:38:58.428274 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:38:58.428274 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:58.428274 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:38:58.428274 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:58.428274 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:38:58.428274 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:38:58.428274 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:58.429108 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:38:58.429108 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:58.429108 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:38:58.429108 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:38:58.429108 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:38:58.429108 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:38:58.429108 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:38:58.429108 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:38:58.429536 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:58.429803 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:38:58.429803 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:58.429803 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:38:58.429803 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:38:58.429803 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:58.429803 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:38:58.429803 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:58.429803 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:38:58.430594 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:58.430594 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:38:58.430594 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:38:58.430594 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:38:58.430594 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:58.430594 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:38:58.430594 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:58.430594 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:38:58.430594 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:58.430594 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:38:58.430594 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:58.430594 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:38:58.430594 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:58.430594 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:38:58.430594 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:38:58.431006 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:58.431006 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:38:58.431006 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:58.431006 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:38:58.431006 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:38:58.431006 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:58.431006 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:38:58.431006 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:58.431006 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:38:58.431426 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:58.431426 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:38:58.431426 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:58.431426 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:38:58.431426 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:38:58.431426 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:38:58.431426 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:38:58.431426 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:38:58.431426 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:38:58.431426 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:38:58.431426 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:58.431426 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:58.431788 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:58.431788 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:38:58.431788 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:58.431788 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:58.431788 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:58.431788 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:38:58.431788 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:58.431788 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:38:58.431788 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:38:58.431788 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:58.431788 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:38:58.431788 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:58.431788 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:38:58.431788 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:38:58.432149 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:58.432149 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:38:58.432149 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:58.432149 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:38:58.432434 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:58.432434 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:38:58.432434 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:38:58.432434 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:38:58.432434 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:38:58.432434 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:38:58.432434 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:58.432434 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:38:58.432434 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:38:58.432434 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:58.432434 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:38:58.432434 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:38:58.432434 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:58.432434 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:58.432434 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:38:58.432434 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:58.432865 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:38:58.432865 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:38:58.432865 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:38:58.432865 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:38:58.432865 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:38:58.432865 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:58.432865 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:58.432865 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:58.432865 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:38:58.432865 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:58.432865 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:58.432865 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:58.432865 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:38:58.432865 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:58.432865 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:38:58.432865 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:38:58.432865 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:58.432865 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:38:58.433373 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:58.433373 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:38:58.433373 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:38:58.433373 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:58.433373 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:38:58.433373 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:58.433373 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:38:58.435569 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:38:58.435846 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:38:58.435940 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:38:58.436077 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:38:58.436425 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:38:58.436641 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:58.437135 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:38:58.437135 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:58.437135 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:38:58.437135 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:38:58.437135 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:58.437135 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:38:58.437135 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:58.437135 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:38:58.437869 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:58.437869 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:38:58.437869 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:38:58.437869 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:38:58.437869 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:58.437869 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:38:58.437869 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:58.437869 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:38:58.437869 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:58.437869 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:38:58.437869 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:58.437869 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:38:58.437869 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:58.437869 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:38:58.437869 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:38:58.438308 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:58.438308 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:38:58.438308 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:58.438308 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:38:58.438308 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:38:58.438308 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:58.438308 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:38:58.438308 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:58.438308 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:38:58.438770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:58.438770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:38:58.438770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:58.438770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:38:58.438770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:38:58.438770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:38:58.438770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:38:58.438770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:38:58.438770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:38:58.438770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:58.438770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:58.439082 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:58.439082 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:38:58.439082 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:58.439082 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:58.439082 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:58.439082 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:38:58.439082 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:58.439082 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:38:58.439082 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:38:58.439082 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:58.439082 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:38:58.439082 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:58.439082 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:38:58.439082 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:38:58.439460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:58.439460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:38:58.439460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:58.439460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:38:58.439810 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:58.439810 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:38:58.439810 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:38:58.439810 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:38:58.439810 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:38:58.439810 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:38:58.439810 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:58.439810 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:38:58.439810 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:38:58.439810 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:58.439810 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:38:58.439810 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:38:58.439810 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:58.439810 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:58.439810 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:38:58.439810 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:58.439810 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:38:58.440274 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:38:58.440274 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:38:58.440274 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:38:58.440274 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:58.440274 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:58.440274 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:58.440274 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:38:58.440274 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:58.440274 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:58.440274 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:58.440274 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:38:58.440274 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:58.440274 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:38:58.440274 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:38:58.440274 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:58.440274 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:38:58.440274 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:58.440274 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:38:58.440274 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:38:58.440799 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:58.440799 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:38:58.440799 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:58.440799 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:38:58.441794 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:38:58.442066 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:38:58.442217 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:38:58.442385 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:38:58.442791 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:38:58.443123 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:38:58.443301 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:58.443971 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:38:58.443971 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:58.443971 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:38:58.443971 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:38:58.443971 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:58.443971 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:38:58.443971 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:58.443971 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:38:58.444789 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:58.444789 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:38:58.444789 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:58.444789 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:38:58.444789 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:38:58.444789 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:38:58.444789 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:38:58.444789 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:58.444789 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:58.444789 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:58.444789 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:38:58.444789 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:58.445144 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:38:58.445144 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:58.445144 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:38:58.445144 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:58.445144 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:38:58.445144 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:38:58.445144 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:58.445144 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:38:58.445144 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:58.445144 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:38:58.445144 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:38:58.445144 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:58.445144 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:38:58.445495 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:58.445495 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:38:58.445754 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:58.445754 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:38:58.445754 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:38:58.445754 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:38:58.445754 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:38:58.445754 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:38:58.445754 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:58.445754 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:38:58.445754 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:38:58.445754 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:58.445754 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:38:58.445754 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:38:58.445754 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:38:58.445754 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:38:58.445754 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:38:58.445754 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:38:58.446218 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:38:58.446218 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:38:58.446218 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:58.446218 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:58.446218 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:38:58.446218 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:38:58.446218 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:38:58.446218 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:38:58.446218 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:38:58.446218 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:38:58.446218 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:38:58.446218 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:38:58.446218 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:38:58.446218 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:38:58.446218 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:38:58.446218 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:38:58.446218 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:38:58.446218 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:38:58.446218 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:38:58.446726 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:38:58.446726 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:38:58.446726 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:39:02.956838 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:39:02.957904 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:39:02.957904 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:39:02.958731 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:39:02.992491 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:39:02.992837 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:02.993720 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:39:02.993720 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:02.993720 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:39:02.993720 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:39:02.993720 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:02.993720 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:39:02.993720 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:02.993720 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:39:02.994967 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.036 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:02.994967 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:39:02.994967 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:02.994967 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:39:02.994967 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:39:02.994967 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:39:02.994967 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:39:02.994967 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:02.994967 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:02.994967 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:02.994967 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:39:02.994967 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:02.995598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:02.995598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:02.995598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:39:02.995598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:02.995598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:39:02.995598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:39:02.995598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:02.995598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:39:02.995598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:02.995598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:39:02.995598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:39:02.995598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:02.995598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:39:02.996418 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:02.996418 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:39:02.996418 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:02.996418 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:39:02.996418 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:39:02.996418 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:39:02.996418 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:39:02.996418 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:39:02.996418 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:02.996418 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:39:02.996418 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:39:02.996418 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:02.996418 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:39:02.996418 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:39:02.996418 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:02.997455 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:02.997455 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:39:02.997455 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:02.997455 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:39:02.997455 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:39:02.997455 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:02.997455 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:02.997455 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:02.997455 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:39:02.997455 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:02.997455 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:02.997455 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:02.997455 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:39:02.997455 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:02.997455 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:39:02.997455 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:39:02.997455 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:02.997455 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:39:02.997455 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:02.998217 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:39:02.998217 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:39:02.998217 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:02.998217 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:39:02.998217 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:02.998217 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:39:03.421734 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:39:03.422160 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:39:03.422422 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:39:03.423479 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:39:03.423847 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:03.424801 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:39:03.424801 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:03.424801 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:39:03.424801 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:39:03.424801 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:03.424801 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:39:03.424801 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:03.424801 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:39:03.426558 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:03.426558 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:39:03.426558 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:03.426558 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:39:03.426558 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:39:03.426558 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:39:03.426558 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:39:03.426558 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:03.426558 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:03.426558 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:03.426558 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:39:03.426558 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:03.427117 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:03.427117 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:03.427117 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:39:03.427117 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:03.427117 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:39:03.427117 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:39:03.427117 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:03.427117 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:39:03.427117 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:03.427117 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:39:03.427117 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:39:03.427117 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:03.427117 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:39:03.427719 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:03.427719 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:39:03.428520 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:03.428520 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:39:03.428520 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:39:03.428520 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:39:03.428520 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:39:03.428520 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:39:03.428520 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:03.428520 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:39:03.428520 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:39:03.428520 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:03.428520 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:39:03.428520 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:39:03.428520 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:03.428520 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:03.428520 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:39:03.428520 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:03.429204 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:39:03.429204 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:39:03.429204 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:03.429204 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:03.429204 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:03.429204 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:39:03.429204 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:03.429204 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:03.429204 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:03.429204 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:39:03.429204 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:03.429204 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:39:03.429204 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:39:03.429204 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:03.429204 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:39:03.429204 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:03.429204 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:39:03.429204 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:39:03.429204 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:03.430009 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:39:03.430009 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:03.430009 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:39:03.430009 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:39:03.430009 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:39:03.430009 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:39:03.430009 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:39:03.430384 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:39:03.430526 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:03.431310 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:39:03.431310 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:03.431310 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:39:03.431310 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:39:03.431310 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:03.431310 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:39:03.431310 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:03.431310 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:39:03.432766 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:03.432766 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:39:03.432766 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:39:03.432766 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:39:03.432766 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:03.432766 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:39:03.432766 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:03.432766 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:39:03.432766 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:03.432766 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:39:03.432766 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:03.432766 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:39:03.432766 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:03.432766 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:39:03.432766 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:39:03.433231 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:03.433231 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:39:03.433231 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:03.433231 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:39:03.433231 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:39:03.433231 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:03.433231 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:39:03.433231 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:03.433231 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:39:03.434302 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:03.434302 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:39:03.434302 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:03.434302 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:39:03.434302 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:39:03.434302 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:39:03.434302 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:39:03.434302 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:39:03.434302 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:39:03.434302 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:39:03.434302 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:03.434302 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:03.434741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:03.434741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:39:03.434741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:03.434741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:03.434741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:03.434741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:39:03.434741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:03.434741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:39:03.434741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:39:03.434741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:03.434741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:39:03.434741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:03.434741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:39:03.434741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:39:03.435149 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:03.435149 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:39:03.435149 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:03.435149 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:39:03.436253 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:03.436253 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:39:03.436253 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:39:03.436253 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:39:03.436253 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:39:03.436253 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:39:03.436253 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:03.436253 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:39:03.436253 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:39:03.436253 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:03.436253 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:39:03.436253 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:39:03.436253 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:03.436253 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:03.436253 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:39:03.436253 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:03.436751 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:39:03.436751 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:39:03.436751 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:39:03.436751 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:39:03.436751 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:39:03.436751 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:03.436751 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:03.436751 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:03.436751 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:39:03.436751 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:03.436751 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:03.436751 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:03.436751 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:39:03.436751 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:03.436751 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:39:03.436751 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:39:03.436751 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:03.436751 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:39:03.437267 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:03.437267 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:39:03.437267 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:39:03.437267 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:03.437267 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:39:03.437267 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:03.437267 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:39:03.437267 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:39:03.437267 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:39:03.437267 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:39:03.437618 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:39:03.437618 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:39:03.437980 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:03.438727 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:39:03.438727 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:03.438727 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:39:03.438727 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:39:03.438727 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:03.438727 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:39:03.438727 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:03.438727 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:39:03.439482 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:03.439482 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:39:03.439482 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:39:03.439482 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:39:03.439482 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:03.439482 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:39:03.439482 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:03.439482 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:39:03.439482 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:03.439482 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:39:03.439482 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:03.439482 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:39:03.439482 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:03.439482 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:39:03.439482 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:39:03.439943 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:03.439943 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:39:03.439943 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:03.439943 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:39:03.439943 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:39:03.439943 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:03.439943 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:39:03.439943 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:03.439943 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:39:03.440436 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:03.440436 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:39:03.440436 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:03.440436 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:39:03.440436 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:39:03.440436 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:39:03.440436 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:39:03.440436 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:39:03.440436 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:39:03.440436 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:03.440436 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:03.440787 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:03.440787 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:39:03.440787 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:03.440787 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:03.440787 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:03.440787 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:39:03.440787 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:03.440787 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:39:03.440787 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:39:03.440787 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:03.440787 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:39:03.440787 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:03.440787 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:39:03.440787 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:39:03.441186 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:03.441186 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:39:03.441186 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:03.441186 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:39:03.441544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:03.441544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:39:03.441544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:39:03.441544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:39:03.441544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:39:03.441544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:39:03.441544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:03.441544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:39:03.441544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:39:03.441544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:03.441544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:39:03.441544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:39:03.441544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:03.441544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:03.441544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:39:03.441544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:03.441544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:39:03.442055 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:39:03.442055 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:39:03.442055 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:39:03.442055 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:03.442055 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:03.442055 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:03.442055 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:39:03.442055 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:03.442055 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:03.442055 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:03.442055 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:39:03.442055 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:03.442055 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:39:03.442055 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:39:03.442055 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:03.442055 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:39:03.442055 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:03.442055 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:39:03.442055 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:39:03.442607 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:03.442607 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:39:03.442607 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:03.442607 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:39:03.442607 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:39:03.442607 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:39:03.442813 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:39:03.442813 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:39:03.443238 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:39:03.443627 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:39:03.443878 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:03.444728 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:39:03.444728 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:03.444728 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:39:03.444728 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:39:03.444728 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:03.444728 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:39:03.444728 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:03.444728 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:39:03.445583 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:03.445583 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:39:03.445583 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:03.445583 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:39:03.445583 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:39:03.445583 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:39:03.445583 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:39:03.445583 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:03.445583 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:03.445583 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:03.445583 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:39:03.445583 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:03.446027 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:03.446027 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:03.446027 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:39:03.446027 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:03.446027 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:39:03.446027 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:39:03.446027 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:03.446027 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:39:03.446027 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:03.446027 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:39:03.446027 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:39:03.446027 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:03.446027 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:39:03.446421 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:03.446421 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:39:03.446589 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:03.446589 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:39:03.446589 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:39:03.446589 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:39:03.446589 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:39:03.446589 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:39:03.446589 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:03.446589 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:39:03.446589 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:39:03.446589 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:03.446589 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:39:03.446589 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:39:03.446589 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:03.446589 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:03.446589 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:39:03.446589 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:03.447068 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:39:03.447068 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:39:03.447068 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:03.447068 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:03.447068 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:03.447068 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:39:03.447068 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:03.447068 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:03.447068 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:03.447068 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:39:03.447068 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:03.447068 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:39:03.447068 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:39:03.447068 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:03.447068 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:39:03.447068 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:03.447068 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:39:03.447068 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:39:03.447068 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:03.447617 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:39:03.447617 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:03.447617 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:39:07.957541 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:39:07.958713 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:39:07.958713 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:39:07.959723 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:39:08.001075 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:39:08.001535 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:08.002386 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:39:08.002386 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:08.002386 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:39:08.002386 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:39:08.002386 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:08.002386 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:39:08.002386 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:08.002386 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:39:08.003167 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.044 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:08.003167 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:39:08.003167 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:08.003167 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:39:08.003167 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:39:08.003167 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:39:08.003167 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:39:08.003167 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:08.003167 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:08.003167 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:08.003167 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:39:08.003167 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:08.003929 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:08.003929 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:08.003929 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:39:08.003929 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:08.003929 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:39:08.003929 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:39:08.003929 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:08.003929 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:39:08.003929 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:08.003929 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:39:08.003929 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:39:08.003929 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:08.003929 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:39:08.004639 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:08.004639 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:39:08.004639 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:08.004639 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:39:08.004639 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:39:08.004639 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:39:08.004639 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:39:08.004639 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:39:08.004639 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:08.004639 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:39:08.004639 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:39:08.004639 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:08.004639 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:39:08.004639 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:39:08.004639 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:08.005366 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:08.005366 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:39:08.005366 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:08.005366 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:39:08.005366 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:39:08.005366 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:08.005366 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:08.005366 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:08.005366 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:39:08.005366 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:08.005366 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:08.005366 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:08.005366 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:39:08.005366 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:08.005366 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:39:08.005366 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:39:08.005366 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:08.005366 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:39:08.005366 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:08.006516 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:39:08.006516 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:39:08.006516 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:08.006516 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:39:08.006516 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:08.006516 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:39:08.422189 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:39:08.422906 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:39:08.423167 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:39:08.424302 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:39:08.424669 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:08.425585 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:39:08.425585 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:08.425585 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:39:08.425585 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:39:08.425585 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:08.425585 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:39:08.425585 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:08.425585 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:39:08.426838 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:08.426838 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:39:08.426838 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:08.426838 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:39:08.426838 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:39:08.426838 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:39:08.426838 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:39:08.426838 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:08.426838 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:08.426838 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:08.426838 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:39:08.426838 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:08.427422 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:08.427422 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:08.427422 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:39:08.427422 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:08.427422 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:39:08.427422 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:39:08.427422 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:08.427422 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:39:08.427422 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:08.427422 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:39:08.427422 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:39:08.427422 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:08.427422 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:39:08.427923 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:08.427923 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:39:08.428038 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:08.428038 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:39:08.428038 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:39:08.428038 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:39:08.428038 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:39:08.428038 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:39:08.428038 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:08.428038 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:39:08.428038 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:39:08.428038 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:08.428038 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:39:08.428038 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:39:08.428038 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:08.428038 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:08.428038 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:39:08.428038 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:08.428726 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:39:08.428726 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:39:08.428726 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:08.428726 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:08.428726 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:08.428726 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:39:08.428726 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:08.428726 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:08.428726 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:08.428726 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:39:08.428726 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:08.428726 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:39:08.428726 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:39:08.428726 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:08.428726 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:39:08.428726 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:08.428726 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:39:08.428726 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:39:08.428726 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:08.429550 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:39:08.429550 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:08.429550 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:39:08.429550 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:39:08.430012 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:39:08.430107 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:39:08.430235 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:39:08.430889 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:39:08.431186 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:08.431902 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:39:08.431902 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:08.431902 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:39:08.431902 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:39:08.431902 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:08.431902 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:39:08.431902 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:08.431902 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:39:08.432759 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:08.432759 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:39:08.432759 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:39:08.432759 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:39:08.432759 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:08.432759 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:39:08.432759 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:08.432759 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:39:08.432759 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:08.432759 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:39:08.432759 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:08.432759 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:39:08.432759 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:08.432759 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:39:08.432759 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:39:08.433156 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:08.433156 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:39:08.433156 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:08.433156 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:39:08.433156 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:39:08.433156 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:08.433156 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:39:08.433156 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:08.433156 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:39:08.433702 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:08.433702 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:39:08.433702 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:08.433702 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:39:08.433702 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:39:08.433702 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:39:08.433702 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:39:08.433702 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:39:08.433702 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:39:08.433702 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:39:08.433702 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:08.433702 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:08.434025 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:08.434025 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:39:08.434025 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:08.434025 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:08.434025 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:08.434025 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:39:08.434025 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:08.434025 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:39:08.434025 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:39:08.434025 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:08.434025 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:39:08.434025 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:08.434025 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:39:08.434025 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:39:08.434385 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:08.434385 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:39:08.434385 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:08.434385 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:39:08.434982 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:08.434982 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:39:08.434982 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:39:08.434982 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:39:08.434982 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:39:08.434982 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:39:08.434982 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:08.434982 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:39:08.434982 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:39:08.434982 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:08.434982 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:39:08.434982 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:39:08.434982 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:08.434982 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:08.434982 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:39:08.434982 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:08.435949 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:39:08.435949 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:39:08.435949 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:39:08.435949 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:39:08.435949 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:39:08.435949 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:08.435949 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:08.435949 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:08.435949 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:39:08.435949 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:08.435949 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:08.435949 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:08.435949 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:39:08.435949 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:08.435949 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:39:08.435949 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:39:08.435949 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:08.435949 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:39:08.436424 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:08.436424 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:39:08.436424 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:39:08.436424 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:08.436424 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:39:08.436424 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:08.436424 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:39:08.437045 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:39:08.437811 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:39:08.437960 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:39:08.438131 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:39:08.438774 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:39:08.439136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:08.439982 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:39:08.439982 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:08.439982 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:39:08.439982 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:39:08.439982 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:08.439982 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:39:08.439982 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:08.439982 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:39:08.440932 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:08.440932 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:39:08.440932 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:39:08.440932 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:39:08.440932 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:08.440932 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:39:08.440932 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:08.440932 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:39:08.440932 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:08.440932 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:39:08.440932 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:08.440932 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:39:08.440932 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:08.440932 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:39:08.440932 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:39:08.441333 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:08.441333 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:39:08.441333 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:08.441333 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:39:08.441333 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:39:08.441333 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:08.441333 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:39:08.441333 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:08.441333 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:39:08.441937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:08.441937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:39:08.441937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:08.441937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:39:08.441937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:39:08.441937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:39:08.441937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:39:08.441937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:39:08.441937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:39:08.441937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:08.441937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:08.442438 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:08.442438 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:39:08.442438 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:08.442438 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:08.442438 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:08.442438 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:39:08.442438 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:08.442438 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:39:08.442438 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:39:08.442438 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:08.442438 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:39:08.442438 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:08.442438 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:39:08.442438 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:39:08.442839 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:08.442839 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:39:08.442839 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:08.442839 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:39:08.443339 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:08.443339 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:39:08.443339 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:39:08.443339 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:39:08.443339 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:39:08.443339 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:39:08.443339 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:08.443339 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:39:08.443339 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:39:08.443339 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:08.443339 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:39:08.443339 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:39:08.443339 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:08.443339 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:08.443339 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:39:08.443339 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:08.443339 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:39:08.443968 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:39:08.443968 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:39:08.443968 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:39:08.443968 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:08.443968 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:08.443968 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:08.443968 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:39:08.443968 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:08.443968 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:08.443968 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:08.443968 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:39:08.443968 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:08.443968 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:39:08.443968 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:39:08.443968 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:08.443968 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:39:08.443968 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:08.443968 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:39:08.443968 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:39:08.444576 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:08.444576 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:39:08.444576 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:08.444576 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:39:08.444576 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:39:08.444576 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:39:08.444576 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:39:08.444576 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:39:08.444839 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:39:08.444984 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:39:08.445155 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:08.445906 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:39:08.445906 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:08.445906 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:39:08.445906 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:39:08.445906 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:08.445906 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:39:08.445906 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:08.445906 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:39:08.446937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:08.446937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:39:08.446937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:08.446937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:39:08.446937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:39:08.446937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:39:08.446937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:39:08.446937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:08.446937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:08.446937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:08.446937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:39:08.446937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:08.447278 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:08.447278 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:08.447278 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:39:08.447278 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:08.447278 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:39:08.447278 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:39:08.447278 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:08.447278 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:39:08.447278 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:08.447278 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:39:08.447278 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:39:08.447278 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:08.447278 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:39:08.447674 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:08.447674 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:39:08.447978 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:08.447978 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:39:08.447978 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:39:08.447978 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:39:08.447978 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:39:08.447978 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:39:08.447978 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:08.447978 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:39:08.447978 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:39:08.447978 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:08.447978 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:39:08.447978 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:39:08.447978 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:08.447978 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:08.447978 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:39:08.447978 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:08.448436 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:39:08.448436 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:39:08.448436 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:08.448436 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:08.448436 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:08.448436 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:39:08.448436 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:08.448436 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:08.448436 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:08.448436 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:39:08.448436 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:08.448436 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:39:08.448436 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:39:08.448436 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:08.448436 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:39:08.448436 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:08.448436 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:39:08.448436 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:39:08.448436 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:08.448951 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:39:08.448951 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:08.448951 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:39:12.958463 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:39:12.959508 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:39:12.959508 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:39:12.959987 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:39:12.986147 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:39:12.986647 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:12.987748 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:39:12.987748 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:12.987748 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:39:12.987748 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:39:12.987748 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:12.987748 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:39:12.987748 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:12.987748 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:39:12.988299 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.028 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:12.988299 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:39:12.988299 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:12.988299 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:39:12.988299 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:39:12.988299 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:39:12.988299 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:39:12.988299 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:12.988299 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:12.988299 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:12.988299 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:39:12.988299 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:12.988880 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:12.988880 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:12.988880 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:39:12.988880 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:12.988880 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:39:12.988880 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:39:12.988880 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:12.988880 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:39:12.988880 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:12.988880 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:39:12.988880 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:39:12.988880 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:12.988880 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:39:12.989442 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:12.989442 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:39:12.989442 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:12.989442 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:39:12.989442 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:39:12.989442 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:39:12.989442 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:39:12.989442 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:39:12.989442 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:12.989442 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:39:12.989442 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:39:12.989442 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:12.989442 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:39:12.989442 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:39:12.989442 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:12.990080 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:12.990080 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:39:12.990080 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:12.990080 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:39:12.990080 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:39:12.990080 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:12.990080 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:12.990080 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:12.990080 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:39:12.990080 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:12.990080 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:12.990080 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:12.990080 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:39:12.990080 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:12.990080 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:39:12.990080 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:39:12.990080 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:12.990080 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:39:12.990080 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:12.991124 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:39:12.991124 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:39:12.991124 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:12.991124 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:39:12.991124 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:12.991124 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:39:13.422956 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:39:13.423506 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:39:13.423709 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:39:13.424756 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:39:13.425076 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:13.425937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:39:13.425937 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:13.425937 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:39:13.425937 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:39:13.425937 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:13.425937 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:39:13.425937 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:13.425937 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:39:13.427582 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:13.427582 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:39:13.427582 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:13.427582 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:39:13.427582 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:39:13.427582 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:39:13.427582 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:39:13.427582 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:13.427582 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:13.427582 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:13.427582 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:39:13.427582 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:13.428396 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:13.428396 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:13.428396 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:39:13.428396 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:13.428396 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:39:13.428396 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:39:13.428396 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:13.428396 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:39:13.428396 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:13.428396 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:39:13.428396 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:39:13.428396 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:13.428396 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:39:13.428967 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:13.428967 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:39:13.428967 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:13.428967 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:39:13.428967 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:39:13.428967 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:39:13.428967 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:39:13.428967 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:39:13.428967 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:13.428967 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:39:13.428967 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:39:13.428967 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:13.428967 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:39:13.428967 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:39:13.428967 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:13.429662 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:13.429662 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:39:13.429662 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:13.429662 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:39:13.429662 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:39:13.429662 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:13.429662 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:13.429662 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:13.429662 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:39:13.429662 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:13.429662 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:13.429662 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:13.429662 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:39:13.429662 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:13.429662 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:39:13.429662 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:39:13.429662 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:13.429662 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:39:13.429662 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:13.430474 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:39:13.430474 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:39:13.430474 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:13.430474 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:39:13.430474 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:13.430474 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:39:13.430474 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:39:13.430474 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:39:13.430834 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:39:13.430834 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:39:13.431234 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:39:13.431578 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:13.432335 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:39:13.432335 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:13.432335 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:39:13.432335 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:39:13.432335 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:13.432335 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:39:13.432335 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:13.432335 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:39:13.433151 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:13.433151 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:39:13.433151 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:39:13.433151 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:39:13.433151 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:13.433151 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:39:13.433151 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:13.433151 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:39:13.433151 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:13.433151 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:39:13.433151 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:13.433151 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:39:13.433151 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:13.433151 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:39:13.433151 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:39:13.433594 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:13.433594 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:39:13.433594 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:13.433594 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:39:13.433594 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:39:13.433594 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:13.433594 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:39:13.433594 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:13.433594 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:39:13.433991 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:13.433991 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:39:13.433991 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:13.433991 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:39:13.433991 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:39:13.433991 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:39:13.433991 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:39:13.433991 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:39:13.433991 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:39:13.433991 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:39:13.433991 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:13.433991 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:13.434313 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:13.434313 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:39:13.434313 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:13.434313 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:13.434313 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:13.434313 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:39:13.434313 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:13.434313 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:39:13.434313 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:39:13.434313 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:13.434313 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:39:13.434313 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:13.434313 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:39:13.434313 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:39:13.434806 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:13.434806 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:39:13.434806 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:13.434806 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:39:13.435021 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:13.435021 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:39:13.435021 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:39:13.435021 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:39:13.435021 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:39:13.435021 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:39:13.435021 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:13.435021 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:39:13.435021 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:39:13.435021 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:13.435021 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:39:13.435021 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:39:13.435021 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:13.435021 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:13.435021 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:39:13.435021 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:13.435476 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:39:13.435476 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:39:13.435476 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:39:13.435476 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:39:13.435476 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:39:13.435476 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:13.435476 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:13.435476 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:13.435476 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:39:13.435476 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:13.435476 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:13.435476 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:13.435476 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:39:13.435476 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:13.435476 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:39:13.435476 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:39:13.435476 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:13.435476 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:39:13.435920 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:13.435920 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:39:13.435920 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:39:13.435920 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:13.435920 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:39:13.435920 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:13.435920 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:39:13.437679 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:39:13.437905 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:39:13.438013 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:39:13.438136 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:39:13.438668 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:39:13.439094 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:13.439981 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:39:13.439981 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:13.439981 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:39:13.439981 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:39:13.439981 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:13.439981 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:39:13.439981 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:13.439981 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:39:13.440710 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:13.440710 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:39:13.440710 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:39:13.440710 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:39:13.440710 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:13.440710 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:39:13.440710 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:13.440710 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:39:13.440710 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:13.440710 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:39:13.440710 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:13.440710 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:39:13.440710 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:13.440710 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:39:13.440710 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:39:13.441127 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:13.441127 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:39:13.441127 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:13.441127 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:39:13.441127 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:39:13.441127 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:13.441127 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:39:13.441127 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:13.441127 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:39:13.441549 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:13.441549 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:39:13.441549 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:13.441549 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:39:13.441549 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:39:13.441549 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:39:13.441549 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:39:13.441549 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:39:13.441549 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:39:13.441549 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:13.441549 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:13.441890 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:13.441890 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:39:13.441890 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:13.441890 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:13.441890 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:13.441890 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:39:13.441890 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:13.441890 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:39:13.441890 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:39:13.441890 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:13.441890 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:39:13.441890 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:13.441890 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:39:13.441890 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:39:13.442272 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:13.442272 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:39:13.442272 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:13.442272 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:39:13.442556 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:13.442556 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:39:13.442556 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:39:13.442556 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:39:13.442556 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:39:13.442556 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:39:13.442556 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:13.442556 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:39:13.442556 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:39:13.442556 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:13.442556 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:39:13.442556 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:39:13.442556 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:13.442556 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:13.442556 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:39:13.442556 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:13.442556 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:39:13.443045 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:39:13.443045 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:39:13.443045 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:39:13.443045 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:13.443045 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:13.443045 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:13.443045 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:39:13.443045 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:13.443045 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:13.443045 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:13.443045 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:39:13.443045 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:13.443045 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:39:13.443045 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:39:13.443045 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:13.443045 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:39:13.443045 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:13.443045 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:39:13.443045 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:39:13.443590 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:13.443590 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:39:13.443590 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:13.443590 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:39:13.443920 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:39:13.444145 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:39:13.444244 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:39:13.444424 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:39:13.444917 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:39:13.445294 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:39:13.445533 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:13.446243 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:39:13.446243 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:13.446243 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:39:13.446243 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:39:13.446243 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:13.446243 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:39:13.446243 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:13.446243 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:39:13.447036 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:13.447036 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:39:13.447036 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:13.447036 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:39:13.447036 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:39:13.447036 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:39:13.447036 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:39:13.447036 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:13.447036 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:13.447036 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:13.447036 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:39:13.447036 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:13.447445 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:13.447445 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:13.447445 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:39:13.447445 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:13.447445 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:39:13.447445 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:39:13.447445 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:13.447445 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:39:13.447445 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:13.447445 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:39:13.447445 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:39:13.447445 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:13.447445 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:39:13.447798 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:13.447798 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:39:13.447972 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:13.447972 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:39:13.447972 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:39:13.447972 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:39:13.447972 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:39:13.447972 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:39:13.447972 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:13.447972 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:39:13.447972 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:39:13.447972 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:13.447972 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:39:13.447972 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:39:13.447972 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:13.447972 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:13.447972 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:39:13.447972 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:13.448435 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:39:13.448435 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:39:13.448435 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:13.448435 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:13.448435 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:13.448435 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:39:13.448435 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:13.448435 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:13.448435 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:13.448435 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:39:13.448435 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:13.448435 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:39:13.448435 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:39:13.448435 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:13.448435 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:39:13.448435 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:13.448435 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:39:13.448435 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:39:13.448435 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:13.448959 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:39:13.448959 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:13.448959 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:39:17.959296 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:39:17.960083 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:39:17.960083 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:39:17.961231 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:39:17.996770 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:39:17.997104 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:17.997833 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:39:17.997833 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:17.997833 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:39:17.997833 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:39:17.997833 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:17.997833 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:39:17.997833 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:17.997833 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:39:17.998603 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.038 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:17.998603 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:39:17.998603 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:17.998603 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:39:17.998603 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:39:17.998603 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:39:17.998603 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:39:17.998603 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:17.998603 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:17.998603 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:17.998603 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:39:17.998603 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:17.999217 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:17.999217 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:17.999217 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:39:17.999217 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:17.999217 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:39:17.999217 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:39:17.999217 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:17.999217 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:39:17.999217 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:17.999217 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:39:17.999217 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:39:17.999217 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:17.999217 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:39:17.999581 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:17.999581 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:39:17.999581 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:17.999581 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:39:17.999581 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:39:17.999581 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:39:17.999581 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:39:17.999581 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:39:17.999581 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:17.999581 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:39:17.999581 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:39:17.999581 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:17.999581 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:39:17.999581 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:39:17.999581 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:17.999977 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:17.999977 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:39:17.999977 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:17.999977 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:39:17.999977 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:39:17.999977 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:17.999977 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:17.999977 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:17.999977 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:39:17.999977 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:17.999977 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:17.999977 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:17.999977 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:39:17.999977 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:17.999977 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:39:17.999977 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:39:17.999977 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:17.999977 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:39:17.999977 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:18.000495 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:39:18.000495 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:39:18.000495 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:18.000495 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:39:18.000495 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:18.000495 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:39:18.423810 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:39:18.424185 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:39:18.424459 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:39:18.425659 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:39:18.425985 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:18.426914 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:39:18.426914 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:18.426914 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:39:18.426914 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:39:18.426914 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:18.426914 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:39:18.426914 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:18.426914 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:39:18.428543 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:18.428543 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:39:18.428543 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:18.428543 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:39:18.428543 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:39:18.428543 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:39:18.428543 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:39:18.428543 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:18.428543 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:18.428543 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:18.428543 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:39:18.428543 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:18.429078 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:18.429078 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:18.429078 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:39:18.429078 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:18.429078 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:39:18.429078 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:39:18.429078 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:18.429078 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:39:18.429078 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:18.429078 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:39:18.429078 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:39:18.429078 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:18.429078 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:39:18.429601 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:18.429601 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:39:18.430498 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:18.430498 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:39:18.430498 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:39:18.430498 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:39:18.430498 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:39:18.430498 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:39:18.430498 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:18.430498 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:39:18.430498 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:39:18.430498 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:18.430498 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:39:18.430498 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:39:18.430498 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:18.430498 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:18.430498 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:39:18.430498 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:18.431782 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:39:18.431782 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:39:18.431782 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:18.431782 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:18.431782 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:18.431782 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:39:18.431782 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:18.431782 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:18.431782 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:18.431782 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:39:18.431782 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:18.431782 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:39:18.431782 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:39:18.431782 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:18.431782 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:39:18.431782 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:18.431782 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:39:18.431782 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:39:18.431782 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:18.433232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:39:18.433232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:18.433232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:39:18.433232 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:39:18.433232 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:39:18.433232 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:39:18.433232 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:39:18.433232 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:39:18.433619 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:18.433619 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:39:18.433619 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:18.433619 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:39:18.433619 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:39:18.433619 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:18.433619 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:39:18.433619 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:18.433619 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:39:18.434468 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:18.434468 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:39:18.434468 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:39:18.434468 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:39:18.434468 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:18.434468 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:39:18.434468 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:18.434468 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:39:18.434468 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:18.434468 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:39:18.434468 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:18.434468 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:39:18.434468 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:18.434468 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:39:18.434468 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:39:18.435244 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:18.435244 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:39:18.435244 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:18.435244 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:39:18.435244 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:39:18.435244 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:18.435244 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:39:18.435244 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:18.435244 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:39:18.435244 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:18.435244 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:39:18.435244 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:18.435244 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:39:18.435244 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:39:18.435911 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:39:18.435911 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:39:18.435911 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:39:18.435911 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:39:18.435911 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:39:18.435911 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:18.435911 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:18.435911 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:18.435911 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:39:18.435911 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:18.435911 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:18.435911 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:18.435911 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:39:18.436481 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:18.436481 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:39:18.436481 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:39:18.436481 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:18.436481 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:39:18.436481 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:18.436481 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:39:18.436481 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:39:18.436481 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:18.436481 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:39:18.436481 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:18.436481 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:39:18.437111 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:18.437111 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:39:18.437111 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:39:18.437111 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:39:18.437111 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:39:18.437111 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:39:18.437111 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:18.437111 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:39:18.437111 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:39:18.437111 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:18.437111 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:39:18.437111 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:39:18.437111 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:18.437111 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:18.437111 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:39:18.437111 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:18.437891 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:39:18.437891 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:39:18.437891 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:39:18.437891 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:39:18.437891 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:39:18.437891 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:18.437891 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:18.437891 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:18.437891 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:39:18.437891 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:18.437891 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:18.437891 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:18.437891 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:39:18.437891 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:18.437891 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:39:18.437891 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:39:18.437891 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:18.437891 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:39:18.438747 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:18.438747 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:39:18.438747 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:39:18.438747 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:18.438747 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:39:18.438747 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:18.438747 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:39:18.438747 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:39:18.438747 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:39:18.438747 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:39:18.439239 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:39:18.439239 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:39:18.439239 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:18.439900 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:39:18.439900 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:18.439900 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:39:18.439900 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:39:18.439900 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:18.439900 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:39:18.439900 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:18.439900 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:39:18.440678 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:18.440678 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:39:18.440678 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:39:18.440678 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:39:18.440678 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:18.440678 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:39:18.440678 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:18.440678 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:39:18.440678 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:18.440678 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:39:18.440678 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:18.440678 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:39:18.440678 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:18.440678 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:39:18.440678 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:39:18.441199 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:18.441199 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:39:18.441199 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:18.441199 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:39:18.441199 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:39:18.441199 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:18.441199 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:39:18.441199 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:18.441199 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:39:18.441583 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:18.441583 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:39:18.441583 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:18.441583 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:39:18.441583 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:39:18.441583 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:39:18.441583 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:39:18.441583 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:39:18.441583 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:39:18.441583 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:18.441583 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:18.441910 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:18.441910 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:39:18.441910 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:18.441910 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:18.441910 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:18.441910 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:39:18.441910 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:18.441910 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:39:18.441910 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:39:18.441910 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:18.441910 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:39:18.441910 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:18.441910 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:39:18.441910 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:39:18.442278 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:18.442278 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:39:18.442278 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:18.442278 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:39:18.442619 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:18.442619 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:39:18.442619 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:39:18.442619 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:39:18.442619 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:39:18.442619 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:39:18.442619 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:18.442619 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:39:18.442619 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:39:18.442619 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:18.442619 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:39:18.442619 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:39:18.442619 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:18.442619 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:18.442619 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:39:18.442619 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:18.442619 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:39:18.443088 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:39:18.443088 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:39:18.443088 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:39:18.443088 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:18.443088 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:18.443088 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:18.443088 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:39:18.443088 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:18.443088 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:18.443088 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:18.443088 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:39:18.443088 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:18.443088 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:39:18.443088 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:39:18.443088 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:18.443088 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:39:18.443088 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:18.443088 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:39:18.443088 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:39:18.443611 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:18.443611 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:39:18.443611 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:18.443611 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:39:18.444229 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:39:18.444434 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:39:18.444578 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:39:18.444727 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:39:18.445169 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:39:18.445525 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:39:18.445701 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:18.446371 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:39:18.446371 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:18.446371 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:39:18.446371 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:39:18.446371 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:18.446371 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:39:18.446371 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:18.446371 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:39:18.447177 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:18.447177 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:39:18.447177 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:18.447177 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:39:18.447177 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:39:18.447177 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:39:18.447177 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:39:18.447177 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:18.447177 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:18.447177 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:18.447177 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:39:18.447177 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:18.447648 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:18.447648 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:18.447648 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:39:18.447648 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:18.447648 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:39:18.447648 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:39:18.447648 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:18.447648 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:39:18.447648 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:18.447648 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:39:18.447648 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:39:18.447648 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:18.447648 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:39:18.448132 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:18.448132 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:39:18.448132 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:18.448132 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:39:18.448132 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:39:18.448132 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:39:18.448132 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:39:18.448132 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:39:18.448132 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:18.448132 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:39:18.448132 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:39:18.448132 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:18.448132 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:39:18.448132 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:39:18.448132 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:18.448732 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:18.448732 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:39:18.448732 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:18.448732 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:39:18.448732 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:39:18.448732 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:18.448732 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:18.448732 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:18.448732 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:39:18.448732 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:18.448732 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:18.448732 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:18.448732 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:39:18.448732 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:18.448732 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:39:18.448732 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:39:18.448732 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:18.448732 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:39:18.448732 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:18.449535 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:39:18.449535 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:39:18.449535 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:18.449535 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:39:18.449535 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:18.449535 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:39:22.960185 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:39:22.961109 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:39:22.961186 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:39:22.962632 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:39:22.998092 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:39:22.998669 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:22.999626 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:39:22.999626 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:22.999626 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:39:22.999626 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:39:22.999626 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:22.999626 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:39:22.999626 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:22.999626 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:39:23.000583 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.038 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:23.000583 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:39:23.000583 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:23.000583 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:39:23.000583 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:39:23.000583 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:39:23.000583 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:39:23.000583 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:23.000583 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:23.000583 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:23.000583 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:39:23.000583 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:23.001229 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:23.001229 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:23.001229 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:39:23.001229 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:23.001229 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:39:23.001229 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:39:23.001229 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:23.001229 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:39:23.001229 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:23.001229 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:39:23.001229 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:39:23.001229 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:23.001229 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:39:23.001895 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:23.001895 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:39:23.001895 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:23.001895 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:39:23.001895 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:39:23.001895 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:39:23.001895 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:39:23.001895 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:39:23.001895 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:23.001895 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:39:23.001895 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:39:23.001895 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:23.001895 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:39:23.001895 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:39:23.001895 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:23.002664 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:23.002664 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:39:23.002664 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:23.002664 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:39:23.002664 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:39:23.002664 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:23.002664 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:23.002664 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:23.002664 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:39:23.002664 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:23.002664 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:23.002664 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:23.002664 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:39:23.002664 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:23.002664 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:39:23.002664 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:39:23.002664 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:23.002664 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:39:23.002664 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:23.003572 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:39:23.003572 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:39:23.003572 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:23.003572 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:39:23.003572 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:23.003572 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:39:23.424608 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:39:23.425394 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:39:23.425601 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:39:23.426568 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:39:23.426755 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:23.427440 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:39:23.427440 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:23.427440 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:39:23.427440 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:39:23.427440 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:23.427440 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:39:23.427440 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:23.427440 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:39:23.428958 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:23.428958 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:39:23.428958 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:23.428958 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:39:23.428958 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:39:23.428958 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:39:23.428958 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:39:23.428958 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:23.428958 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:23.428958 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:23.428958 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:39:23.428958 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:23.429635 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:23.429635 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:23.429635 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:39:23.429635 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:23.429635 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:39:23.429635 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:39:23.429635 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:23.429635 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:39:23.429635 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:23.429635 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:39:23.429635 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:39:23.429635 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:23.429635 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:39:23.430255 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:23.430255 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:39:23.430841 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:23.430841 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:39:23.430841 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:39:23.430841 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:39:23.430841 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:39:23.430841 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:39:23.430841 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:23.430841 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:39:23.430841 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:39:23.430841 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:23.430841 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:39:23.430841 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:39:23.430841 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:23.430841 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:23.430841 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:39:23.430841 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:23.431611 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:39:23.431611 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:39:23.431611 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:23.431611 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:23.431611 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:23.431611 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:39:23.431611 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:23.431611 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:23.431611 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:23.431611 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:39:23.431611 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:23.431611 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:39:23.431611 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:39:23.431611 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:23.431611 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:39:23.431611 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:23.431611 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:39:23.431611 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:39:23.431611 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:23.432411 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:39:23.432411 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:23.432411 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:39:23.432411 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:39:23.432411 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:39:23.432411 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:39:23.432411 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:39:23.432411 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:39:23.432795 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:23.433080 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:39:23.433080 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:23.433080 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:39:23.433080 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:39:23.433080 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:23.433080 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:39:23.433080 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:23.433080 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:39:23.434393 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:23.434393 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:39:23.434393 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:39:23.434393 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:39:23.434393 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:23.434393 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:39:23.434393 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:23.434393 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:39:23.434393 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:23.434393 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:39:23.434393 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:23.434393 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:39:23.434393 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:23.434393 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:39:23.434393 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:39:23.435168 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:23.435168 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:39:23.435168 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:23.435168 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:39:23.435168 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:39:23.435168 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:23.435168 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:39:23.435168 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:23.435168 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:39:23.435615 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:23.435615 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:39:23.435615 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:23.435615 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:39:23.435615 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:39:23.435615 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:39:23.435615 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:39:23.435615 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:39:23.435615 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:39:23.435615 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:39:23.435615 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:23.435615 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:23.436144 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:23.436144 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:39:23.436144 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:23.436144 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:23.436144 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:23.436144 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:39:23.436144 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:23.436144 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:39:23.436144 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:39:23.436144 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:23.436144 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:39:23.436144 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:23.436144 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:39:23.436144 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:39:23.436713 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:23.436713 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:39:23.436713 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:23.436713 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:39:23.436713 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:23.436713 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:39:23.436713 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:39:23.436713 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:39:23.436713 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:39:23.436713 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:39:23.436713 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:23.436713 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:39:23.436713 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:39:23.436713 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:23.437294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:39:23.437294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:39:23.437294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:23.437294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:23.437294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:39:23.437294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:23.437294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:39:23.437294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:39:23.437294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:39:23.437294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:39:23.437294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:39:23.437294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:23.437294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:23.437294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:23.437294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:39:23.437294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:23.437294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:23.437294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:23.437294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:39:23.438144 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:23.438144 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:39:23.438144 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:39:23.438144 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:23.438144 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:39:23.438144 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:23.438144 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:39:23.438144 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:39:23.438144 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:23.438144 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:39:23.438144 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:23.438144 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:39:23.438669 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:39:23.438927 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:39:23.439004 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:39:23.439151 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:39:23.439638 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:39:23.439981 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:23.440670 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:39:23.440670 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:23.440670 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:39:23.440670 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:39:23.440670 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:23.440670 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:39:23.440670 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:23.440670 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:39:23.441558 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:23.441558 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:39:23.441558 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:39:23.441558 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:39:23.441558 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:23.441558 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:39:23.441558 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:23.441558 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:39:23.441558 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:23.441558 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:39:23.441558 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:23.441558 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:39:23.441558 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:23.441558 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:39:23.441558 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:39:23.441959 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:23.441959 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:39:23.441959 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:23.441959 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:39:23.441959 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:39:23.441959 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:23.441959 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:39:23.441959 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:23.441959 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:39:23.442442 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:23.442442 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:39:23.442442 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:23.442442 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:39:23.442442 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:39:23.442442 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:39:23.442442 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:39:23.442442 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:39:23.442442 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:39:23.442442 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:23.442442 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:23.442796 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:23.442796 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:39:23.442796 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:23.442796 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:23.442796 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:23.442796 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:39:23.442796 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:23.442796 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:39:23.442796 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:39:23.442796 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:23.442796 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:39:23.442796 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:23.442796 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:39:23.442796 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:39:23.443145 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:23.443145 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:39:23.443145 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:23.443145 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:39:23.443504 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:23.443504 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:39:23.443504 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:39:23.443504 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:39:23.443504 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:39:23.443504 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:39:23.443504 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:23.443504 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:39:23.443504 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:39:23.443504 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:23.443504 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:39:23.443504 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:39:23.443504 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:23.443504 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:23.443504 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:39:23.443504 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:23.443504 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:39:23.443950 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:39:23.443950 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:39:23.443950 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:39:23.443950 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:23.443950 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:23.443950 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:23.443950 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:39:23.443950 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:23.443950 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:23.443950 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:23.443950 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:39:23.443950 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:23.443950 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:39:23.443950 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:39:23.443950 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:23.443950 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:39:23.443950 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:23.443950 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:39:23.443950 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:39:23.444440 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:23.444440 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:39:23.444440 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:23.444440 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:39:23.444618 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:39:23.444928 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:39:23.445048 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:39:23.445274 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:39:23.445665 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:39:23.445999 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:39:23.446176 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:23.446937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:39:23.446937 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:23.446937 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:39:23.446937 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:39:23.446937 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:23.446937 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:39:23.446937 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:23.446937 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:39:23.447888 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:23.447888 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:39:23.447888 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:23.447888 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:39:23.447888 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:39:23.447888 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:39:23.447888 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:39:23.447888 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:23.447888 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:23.447888 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:23.447888 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:39:23.447888 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:23.448231 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:23.448231 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:23.448231 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:39:23.448231 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:23.448231 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:39:23.448231 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:39:23.448231 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:23.448231 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:39:23.448231 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:23.448231 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:39:23.448231 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:39:23.448231 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:23.448231 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:39:23.448570 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:23.448570 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:39:23.448881 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:23.448881 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:39:23.448881 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:39:23.448881 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:39:23.448881 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:39:23.448881 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:39:23.448881 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:23.448881 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:39:23.448881 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:39:23.448881 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:23.448881 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:39:23.448881 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:39:23.448881 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:23.448881 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:23.448881 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:39:23.448881 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:23.449299 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:39:23.449299 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:39:23.449299 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:23.449299 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:23.449299 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:23.449299 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:39:23.449299 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:23.449299 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:23.449299 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:23.449299 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:39:23.449299 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:23.449299 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:39:23.449299 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:39:23.449299 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:23.449299 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:39:23.449299 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:23.449299 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:39:23.449299 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:39:23.449299 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:23.449847 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:39:23.449847 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:23.449847 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:39:27.960898 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:39:27.961702 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:39:27.961702 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:39:27.962375 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:39:27.991910 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:39:27.992018 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:27.992616 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:39:27.992616 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:27.992616 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:39:27.992616 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:39:27.992616 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:27.992616 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:39:27.992616 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:27.992616 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:39:27.993936 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.031 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:27.993936 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:39:27.993936 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:27.993936 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:39:27.993936 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:39:27.993936 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:39:27.993936 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:39:27.993936 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:27.993936 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:27.993936 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:27.993936 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:39:27.993936 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:27.994544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:27.994544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:27.994544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:39:27.994544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:27.994544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:39:27.994544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:39:27.994544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:27.994544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:39:27.994544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:27.994544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:39:27.994544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:39:27.994544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:27.994544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:39:27.995369 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:27.995369 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:39:27.995574 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:27.995574 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:39:27.995574 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:39:27.995574 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:39:27.995574 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:39:27.995574 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:39:27.995574 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:27.995574 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:39:27.995574 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:39:27.995574 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:27.995574 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:39:27.995574 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:39:27.995574 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:27.995574 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:27.995574 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:39:27.995574 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:27.996569 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:39:27.996569 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:39:27.996569 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:27.996569 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:27.996569 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:27.996569 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:39:27.996569 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:27.996569 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:27.996569 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:27.996569 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:39:27.996569 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:27.996569 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:39:27.996569 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:39:27.996569 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:27.996569 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:39:27.996569 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:27.996569 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:39:27.996569 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:39:27.996569 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:27.997492 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:39:27.997492 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:27.997492 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:39:28.425359 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:39:28.425970 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:39:28.426120 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:39:28.427022 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:39:28.427372 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:28.428205 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:39:28.428205 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:28.428205 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:39:28.428205 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:39:28.428205 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:28.428205 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:39:28.428205 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:28.428205 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:39:28.429743 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:28.429743 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:39:28.429743 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:28.429743 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:39:28.429743 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:39:28.429743 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:39:28.429743 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:39:28.429743 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:28.429743 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:28.429743 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:28.429743 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:39:28.429743 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:28.430547 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:28.430547 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:28.430547 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:39:28.430547 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:28.430547 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:39:28.430547 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:39:28.430547 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:28.430547 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:39:28.430547 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:28.430547 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:39:28.430547 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:39:28.430547 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:28.430547 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:39:28.431033 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:28.431033 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:39:28.431596 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:28.431596 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:39:28.431596 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:39:28.431596 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:39:28.431596 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:39:28.431596 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:39:28.431596 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:28.431596 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:39:28.431596 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:39:28.431596 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:28.431596 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:39:28.431596 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:39:28.431596 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:28.431596 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:28.431596 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:39:28.431596 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:28.433147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:39:28.433147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:39:28.433147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:28.433147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:28.433147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:28.433147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:39:28.433147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:28.433147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:28.433147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:28.433147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:39:28.433147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:28.433147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:39:28.433147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:39:28.433147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:28.433147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:39:28.433147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:28.433147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:39:28.433147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:39:28.433147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:28.434174 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:39:28.434174 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:28.434174 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:39:28.434174 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:39:28.434174 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:39:28.434174 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:39:28.434174 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:39:28.434174 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:39:28.434604 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:28.434604 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:39:28.434604 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:28.434604 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:39:28.434604 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:39:28.434604 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:28.434604 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:39:28.434604 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:28.434604 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:39:28.435514 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:28.435514 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:39:28.435514 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:39:28.435514 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:39:28.435514 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:28.435514 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:39:28.435514 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:28.435514 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:39:28.435514 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:28.435514 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:39:28.435514 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:28.435514 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:39:28.435514 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:28.435514 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:39:28.435514 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:39:28.436994 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:28.436994 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:39:28.436994 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:28.436994 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:39:28.436994 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:39:28.436994 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:28.436994 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:39:28.436994 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:28.436994 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:39:28.436994 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:28.436994 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:39:28.436994 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:28.436994 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:39:28.436994 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:39:28.437904 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:39:28.437904 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:39:28.437904 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:39:28.437904 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:39:28.437904 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:39:28.437904 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:28.437904 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:28.437904 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:28.437904 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:39:28.437904 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:28.437904 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:28.437904 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:28.437904 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:39:28.438477 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:28.438477 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:39:28.438477 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:39:28.438477 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:28.438477 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:39:28.438477 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:28.438477 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:39:28.438477 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:39:28.438477 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:28.438477 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:39:28.438477 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:28.438477 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:39:28.438756 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:28.438756 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:39:28.438756 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:39:28.438756 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:39:28.438756 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:39:28.438756 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:39:28.438756 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:28.438756 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:39:28.438756 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:39:28.438756 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:28.438756 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:39:28.438756 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:39:28.438756 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:28.438756 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:28.438756 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:39:28.438756 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:28.439136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:39:28.439136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:39:28.439136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:39:28.439136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:39:28.439136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:39:28.439136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:28.439136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:28.439136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:28.439136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:39:28.439136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:28.439136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:28.439136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:28.439136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:39:28.439136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:28.439136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:39:28.439136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:39:28.439136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:28.439136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:39:28.439877 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:28.439877 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:39:28.439877 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:39:28.439877 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:28.439877 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:39:28.439877 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:28.439877 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:39:28.439877 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:39:28.439877 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:39:28.439877 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:39:28.440156 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:39:28.440156 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:39:28.440156 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:28.440486 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:39:28.440486 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:28.440486 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:39:28.440486 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:39:28.440486 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:28.440486 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:39:28.440486 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:28.440486 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:39:28.441225 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:28.441225 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:39:28.441225 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:39:28.441225 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:39:28.441225 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:28.441225 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:39:28.441225 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:28.441225 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:39:28.441225 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:28.441225 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:39:28.441225 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:28.441225 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:39:28.441225 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:28.441225 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:39:28.441225 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:39:28.442086 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:28.442086 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:39:28.442086 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:28.442086 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:39:28.442086 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:39:28.442086 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:28.442086 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:39:28.442086 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:28.442086 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:39:28.442460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:28.442460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:39:28.442460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:28.442460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:39:28.442460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:39:28.442460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:39:28.442460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:39:28.442460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:39:28.442460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:39:28.442460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:28.442460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:28.442832 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:28.442832 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:39:28.442832 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:28.442832 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:28.442832 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:28.442832 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:39:28.442832 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:28.442832 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:39:28.442832 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:39:28.442832 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:28.442832 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:39:28.442832 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:28.442832 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:39:28.442832 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:39:28.443158 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:28.443158 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:39:28.443158 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:28.443158 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:39:28.443367 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:28.443367 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:39:28.443367 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:39:28.443367 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:39:28.443367 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:39:28.443367 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:39:28.443367 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:28.443367 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:39:28.443367 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:39:28.443367 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:28.443367 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:39:28.443367 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:39:28.443367 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:28.443367 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:28.443367 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:39:28.443367 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:28.443367 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:39:28.444617 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:39:28.444617 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:39:28.444617 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:39:28.444617 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:28.444617 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:28.444617 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:28.444617 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:39:28.444617 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:28.444617 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:28.444617 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:28.444617 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:39:28.444617 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:28.444617 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:39:28.444617 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:39:28.444617 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:28.444617 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:39:28.444617 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:28.444617 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:39:28.444617 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:39:28.445135 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:28.445135 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:39:28.445135 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:28.445135 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:39:28.445135 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:39:28.445432 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:39:28.445526 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:39:28.445675 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:39:28.445968 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:39:28.446188 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:39:28.446303 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:28.446888 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:39:28.446888 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:28.446888 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:39:28.446888 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:39:28.446888 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:28.446888 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:39:28.446888 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:28.446888 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:39:28.447770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:28.447770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:39:28.447770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:28.447770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:39:28.447770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:39:28.447770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:39:28.447770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:39:28.447770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:28.447770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:28.447770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:28.447770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:39:28.447770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:28.448298 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:28.448298 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:28.448298 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:39:28.448298 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:28.448298 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:39:28.448298 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:39:28.448298 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:28.448298 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:39:28.448298 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:28.448298 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:39:28.448298 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:39:28.448298 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:28.448298 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:39:28.448529 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:28.448529 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:39:28.448912 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:28.448912 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:39:28.448912 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:39:28.448912 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:39:28.448912 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:39:28.448912 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:39:28.448912 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:28.448912 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:39:28.448912 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:39:28.448912 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:28.448912 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:39:28.448912 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:39:28.448912 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:28.448912 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:28.448912 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:39:28.448912 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:28.449203 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:39:28.449203 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:39:28.449203 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:28.449203 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:28.449203 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:28.449203 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:39:28.449203 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:28.449203 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:28.449203 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:28.449203 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:39:28.449203 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:28.449203 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:39:28.449203 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:39:28.449203 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:28.449203 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:39:28.449203 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:28.449203 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:39:28.449203 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:39:28.449203 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:28.449550 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:39:28.449550 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:28.449550 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:39:32.961590 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:39:32.962436 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:39:32.962436 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:39:32.963548 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:39:32.994433 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:39:32.994816 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:32.995760 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:39:32.995760 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:32.995760 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:39:32.995760 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:39:32.995760 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:32.995760 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:39:32.995760 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:32.995760 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:39:32.996565 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.033 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:32.996565 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:39:32.996565 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:32.996565 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:39:32.996565 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:39:32.996565 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:39:32.996565 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:39:32.996565 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:32.996565 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:32.996565 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:32.996565 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:39:32.996565 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:32.997180 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:32.997180 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:32.997180 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:39:32.997180 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:32.997180 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:39:32.997180 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:39:32.997180 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:32.997180 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:39:32.997180 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:32.997180 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:39:32.997180 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:39:32.997180 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:32.997180 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:39:32.997723 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:32.997723 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:39:32.997723 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:32.997723 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:39:32.997723 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:39:32.997723 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:39:32.997723 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:39:32.997723 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:39:32.997723 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:32.997723 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:39:32.997723 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:39:32.997723 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:32.997723 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:39:32.997723 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:39:32.997723 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:32.998956 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:32.998956 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:39:32.998956 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:32.998956 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:39:32.998956 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:39:32.998956 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:32.998956 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:32.998956 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:32.998956 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:39:32.998956 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:32.998956 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:32.998956 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:32.998956 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:39:32.998956 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:32.998956 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:39:32.998956 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:39:32.998956 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:32.998956 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:39:32.998956 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:32.999932 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:39:32.999932 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:39:32.999932 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:32.999932 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:39:32.999932 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:32.999932 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:39:33.426054 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:39:33.426796 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:39:33.426997 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:39:33.427713 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:39:33.427895 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:33.428564 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:39:33.428564 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:33.428564 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:39:33.428564 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:39:33.428564 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:33.428564 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:39:33.428564 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:33.428564 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:39:33.430008 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:33.430008 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:39:33.430008 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:33.430008 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:39:33.430008 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:39:33.430008 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:39:33.430008 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:39:33.430008 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:33.430008 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:33.430008 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:33.430008 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:39:33.430008 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:33.430691 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:33.430691 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:33.430691 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:39:33.430691 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:33.430691 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:39:33.430691 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:39:33.430691 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:33.430691 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:39:33.430691 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:33.430691 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:39:33.430691 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:39:33.430691 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:33.430691 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:39:33.431378 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:33.431378 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:39:33.431512 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:33.431512 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:39:33.431512 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:39:33.431512 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:39:33.431512 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:39:33.431512 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:39:33.431512 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:33.431512 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:39:33.431512 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:39:33.431512 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:33.431512 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:39:33.431512 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:39:33.431512 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:33.431512 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:33.431512 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:39:33.431512 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:33.432250 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:39:33.432250 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:39:33.432250 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:33.432250 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:33.432250 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:33.432250 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:39:33.432250 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:33.432250 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:33.432250 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:33.432250 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:39:33.432250 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:33.432250 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:39:33.432250 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:39:33.432250 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:33.432250 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:39:33.432250 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:33.432250 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:39:33.432250 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:39:33.432250 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:33.433003 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:39:33.433003 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:33.433003 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:39:33.433003 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:39:33.433003 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:39:33.433003 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:39:33.433003 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:39:33.433703 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:39:33.433703 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:33.434188 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:39:33.434188 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:33.434188 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:39:33.434188 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:39:33.434188 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:33.434188 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:39:33.434188 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:33.434188 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:39:33.434844 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:33.434844 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:39:33.434844 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:39:33.434844 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:39:33.434844 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:33.434844 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:39:33.434844 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:33.434844 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:39:33.434844 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:33.434844 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:39:33.434844 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:33.434844 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:39:33.434844 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:33.434844 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:39:33.434844 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:39:33.435666 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:33.435666 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:39:33.435666 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:33.435666 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:39:33.435666 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:39:33.435666 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:33.435666 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:39:33.435666 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:33.435666 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:39:33.436018 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:33.436018 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:39:33.436018 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:33.436018 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:39:33.436018 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:39:33.436018 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:39:33.436018 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:39:33.436018 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:39:33.436018 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:39:33.436018 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:39:33.436018 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:33.436018 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:33.436639 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:33.436639 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:39:33.436639 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:33.436639 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:33.436639 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:33.436639 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:39:33.436639 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:33.436639 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:39:33.436639 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:39:33.436639 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:33.436639 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:39:33.436639 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:33.436639 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:39:33.436639 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:39:33.437154 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:33.437154 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:39:33.437154 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:33.437154 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:39:33.437154 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:33.437154 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:39:33.437154 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:39:33.437154 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:39:33.437154 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:39:33.437154 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:39:33.437154 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:33.437154 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:39:33.437154 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:39:33.437154 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:33.437731 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:39:33.437731 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:39:33.437731 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:33.437731 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:33.437731 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:39:33.437731 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:33.437731 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:39:33.437731 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:39:33.437731 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:39:33.437731 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:39:33.437731 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:39:33.437731 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:33.437731 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:33.437731 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:33.437731 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:39:33.437731 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:33.437731 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:33.437731 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:33.437731 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:39:33.438797 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:33.438797 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:39:33.438797 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:39:33.438797 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:33.438797 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:39:33.438797 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:33.438797 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:39:33.438797 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:39:33.438797 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:33.438797 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:39:33.438797 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:33.438797 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:39:33.439475 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:39:33.439746 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:39:33.439869 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:39:33.440005 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:39:33.440287 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:39:33.440527 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:33.440960 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:39:33.440960 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:33.440960 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:39:33.440960 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:39:33.440960 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:33.440960 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:39:33.440960 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:33.440960 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:39:33.441780 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:33.441780 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:39:33.441780 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:39:33.441780 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:39:33.441780 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:33.441780 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:39:33.441780 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:33.441780 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:39:33.441780 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:33.441780 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:39:33.441780 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:33.441780 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:39:33.441780 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:33.441780 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:39:33.441780 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:39:33.442411 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:33.442411 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:39:33.442411 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:33.442411 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:39:33.442411 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:39:33.442411 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:33.442411 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:39:33.442411 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:33.442411 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:39:33.442855 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:33.442855 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:39:33.442855 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:33.442855 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:39:33.442855 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:39:33.442855 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:39:33.442855 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:39:33.442855 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:39:33.442855 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:39:33.442855 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:33.442855 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:33.443303 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:33.443303 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:39:33.443303 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:33.443303 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:33.443303 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:33.443303 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:39:33.443303 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:33.443303 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:39:33.443303 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:39:33.443303 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:33.443303 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:39:33.443303 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:33.443303 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:39:33.443303 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:39:33.443844 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:33.443844 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:39:33.443844 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:33.443844 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:39:33.444001 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:33.444001 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:39:33.444001 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:39:33.444001 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:39:33.444001 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:39:33.444001 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:39:33.444001 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:33.444001 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:39:33.444001 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:39:33.444001 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:33.444001 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:39:33.444001 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:39:33.444001 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:33.444001 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:33.444001 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:39:33.444001 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:33.444001 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:39:33.444667 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:39:33.444667 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:39:33.444667 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:39:33.444667 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:33.444667 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:33.444667 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:33.444667 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:39:33.444667 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:33.444667 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:33.444667 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:33.444667 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:39:33.444667 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:33.444667 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:39:33.444667 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:39:33.444667 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:33.444667 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:39:33.444667 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:33.444667 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:39:33.444667 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:39:33.445421 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:33.445421 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:39:33.445421 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:33.445421 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:39:33.445577 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:39:33.445784 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:39:33.445919 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:39:33.446124 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:39:33.446474 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:39:33.446687 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:39:33.446793 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:33.447256 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:39:33.447256 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:33.447256 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:39:33.447256 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:39:33.447256 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:33.447256 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:39:33.447256 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:33.447256 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:39:33.448163 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:33.448163 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:39:33.448163 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:33.448163 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:39:33.448163 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:39:33.448163 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:39:33.448163 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:39:33.448163 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:33.448163 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:33.448163 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:33.448163 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:39:33.448163 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:33.448828 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:33.448828 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:33.448828 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:39:33.448828 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:33.448828 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:39:33.448828 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:39:33.448828 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:33.448828 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:39:33.448828 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:33.448828 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:39:33.448828 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:39:33.448828 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:33.448828 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:39:33.449499 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:33.449499 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:39:33.449499 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:33.449499 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:39:33.449499 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:39:33.449499 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:39:33.449499 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:39:33.449499 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:39:33.449499 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:33.449499 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:39:33.449499 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:39:33.449499 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:33.449499 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:39:33.449499 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:39:33.449499 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:33.450282 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:33.450282 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:39:33.450282 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:33.450282 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:39:33.450282 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:39:33.450282 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:33.450282 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:33.450282 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:33.450282 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:39:33.450282 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:33.450282 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:33.450282 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:33.450282 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:39:33.450282 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:33.450282 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:39:33.450282 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:39:33.450282 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:33.450282 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:39:33.450282 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:33.450959 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:39:33.450959 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:39:33.450959 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:33.450959 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:39:33.450959 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:33.450959 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:39:37.962213 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:39:37.962977 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:39:37.963040 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:39:37.964575 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:39:37.999075 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:39:37.999580 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:38.000521 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:39:38.000521 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:38.000521 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:39:38.000521 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:39:38.000521 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:38.000521 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:39:38.000521 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:38.000521 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:39:38.001390 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.037 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:38.001390 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:39:38.001390 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:38.001390 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:39:38.001390 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:39:38.001390 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:39:38.001390 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:39:38.001390 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:38.001390 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:38.001390 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:38.001390 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:39:38.001390 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:38.001907 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:38.001907 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:38.001907 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:39:38.001907 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:38.001907 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:39:38.001907 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:39:38.001907 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:38.001907 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:39:38.001907 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:38.001907 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:39:38.001907 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:39:38.001907 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:38.001907 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:39:38.002482 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:38.002482 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:39:38.002482 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:38.002482 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:39:38.002482 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:39:38.002482 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:39:38.002482 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:39:38.002482 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:39:38.002482 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:38.002482 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:39:38.002482 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:39:38.002482 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:38.002482 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:39:38.002482 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:39:38.002482 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:38.003969 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:38.003969 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:39:38.003969 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:38.003969 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:39:38.003969 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:39:38.003969 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:38.003969 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:38.003969 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:38.003969 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:39:38.003969 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:38.003969 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:38.003969 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:38.003969 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:39:38.003969 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:38.003969 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:39:38.003969 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:39:38.003969 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:38.003969 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:39:38.003969 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:38.006837 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:39:38.006837 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:39:38.006837 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:38.006837 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:39:38.006837 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:38.006837 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:39:38.426876 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:39:38.427449 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:39:38.427651 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:39:38.428698 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:39:38.429009 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:38.429894 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:39:38.429894 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:38.429894 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:39:38.429894 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:39:38.429894 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:38.429894 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:39:38.429894 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:38.429894 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:39:38.431570 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:38.431570 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:39:38.431570 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:38.431570 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:39:38.431570 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:39:38.431570 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:39:38.431570 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:39:38.431570 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:38.431570 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:38.431570 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:38.431570 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:39:38.431570 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:38.432123 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:38.432123 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:38.432123 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:39:38.432123 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:38.432123 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:39:38.432123 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:39:38.432123 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:38.432123 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:39:38.432123 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:38.432123 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:39:38.432123 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:39:38.432123 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:38.432123 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:39:38.433067 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:38.433067 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:39:38.433315 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:38.433315 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:39:38.433315 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:39:38.433315 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:39:38.433315 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:39:38.433315 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:39:38.433315 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:38.433315 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:39:38.433315 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:39:38.433315 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:38.433315 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:39:38.433315 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:39:38.433315 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:38.433315 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:38.433315 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:39:38.433315 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:38.434224 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:39:38.434224 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:39:38.434224 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:38.434224 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:38.434224 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:38.434224 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:39:38.434224 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:38.434224 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:38.434224 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:38.434224 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:39:38.434224 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:38.434224 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:39:38.434224 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:39:38.434224 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:38.434224 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:39:38.434224 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:38.434224 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:39:38.434224 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:39:38.434224 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:38.434868 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:39:38.434868 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:38.434868 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:39:38.434868 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:39:38.434868 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:39:38.434868 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:39:38.434868 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:39:38.435204 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:39:38.435544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:38.436285 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:39:38.436285 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:38.436285 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:39:38.436285 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:39:38.436285 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:38.436285 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:39:38.436285 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:38.436285 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:39:38.437058 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:38.437058 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:39:38.437058 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:39:38.437058 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:39:38.437058 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:38.437058 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:39:38.437058 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:38.437058 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:39:38.437058 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:38.437058 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:39:38.437058 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:38.437058 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:39:38.437058 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:38.437058 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:39:38.437058 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:39:38.437510 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:38.437510 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:39:38.437510 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:38.437510 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:39:38.437510 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:39:38.437510 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:38.437510 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:39:38.437510 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:38.437510 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:39:38.437871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:38.437871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:39:38.437871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:38.437871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:39:38.437871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:39:38.437871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:39:38.437871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:39:38.437871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:39:38.437871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:39:38.437871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:39:38.437871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:38.437871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:38.438296 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:38.438296 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:39:38.438296 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:38.438296 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:38.438296 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:38.438296 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:39:38.438296 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:38.438296 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:39:38.438296 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:39:38.438296 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:38.438296 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:39:38.438296 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:38.438296 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:39:38.438296 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:39:38.438686 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:38.438686 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:39:38.438686 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:38.438686 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:39:38.438861 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:38.438861 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:39:38.438861 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:39:38.438861 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:39:38.438861 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:39:38.438861 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:39:38.438861 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:38.438861 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:39:38.438861 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:39:38.438861 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:38.438861 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:39:38.438861 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:39:38.438861 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:38.438861 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:38.438861 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:39:38.438861 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:38.439264 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:39:38.439264 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:39:38.439264 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:39:38.439264 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:39:38.439264 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:39:38.439264 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:38.439264 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:38.439264 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:38.439264 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:39:38.439264 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:38.439264 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:38.439264 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:38.439264 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:39:38.439264 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:38.439264 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:39:38.439264 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:39:38.439264 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:38.439264 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:39:38.439729 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:38.439729 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:39:38.439729 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:39:38.439729 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:38.439729 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:39:38.439729 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:38.439729 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:39:38.439902 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:39:38.440024 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:39:38.440126 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:39:38.440271 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:39:38.440611 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:39:38.440843 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:38.441339 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:39:38.441339 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:38.441339 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:39:38.441339 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:39:38.441339 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:38.441339 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:39:38.441339 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:38.441339 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:39:38.442059 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:38.442059 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:39:38.442059 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:39:38.442059 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:39:38.442059 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:38.442059 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:39:38.442059 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:38.442059 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:39:38.442059 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:38.442059 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:39:38.442059 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:38.442059 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:39:38.442059 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:38.442059 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:39:38.442059 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:39:38.442358 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:38.442358 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:39:38.442358 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:38.442358 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:39:38.442358 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:39:38.442358 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:38.442358 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:39:38.442358 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:38.442358 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:39:38.443007 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:38.443007 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:39:38.443007 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:38.443007 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:39:38.443007 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:39:38.443007 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:39:38.443007 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:39:38.443007 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:39:38.443007 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:39:38.443007 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:38.443007 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:38.443209 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:38.443209 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:39:38.443209 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:38.443209 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:38.443209 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:38.443209 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:39:38.443209 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:38.443209 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:39:38.443209 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:39:38.443209 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:38.443209 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:39:38.443209 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:38.443209 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:39:38.443209 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:39:38.443465 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:38.443465 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:39:38.443465 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:38.443465 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:39:38.444194 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:38.444194 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:39:38.444194 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:39:38.444194 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:39:38.444194 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:39:38.444194 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:39:38.444194 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:38.444194 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:39:38.444194 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:39:38.444194 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:38.444194 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:39:38.444194 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:39:38.444194 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:38.444194 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:38.444194 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:39:38.444194 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:38.444194 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:39:38.444554 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:39:38.444554 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:39:38.444554 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:39:38.444554 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:38.444554 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:38.444554 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:38.444554 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:39:38.444554 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:38.444554 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:38.444554 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:38.444554 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:39:38.444554 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:38.444554 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:39:38.444554 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:39:38.444554 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:38.444554 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:39:38.444554 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:38.444554 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:39:38.444554 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:39:38.444890 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:38.444890 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:39:38.444890 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:38.444890 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:39:38.445950 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:39:38.446214 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:39:38.446368 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:39:38.446556 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:39:38.446857 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:39:38.447182 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:39:38.447385 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:38.448016 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:39:38.448016 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:38.448016 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:39:38.448016 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:39:38.448016 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:38.448016 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:39:38.448016 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:38.448016 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:39:38.448892 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:38.448892 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:39:38.448892 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:38.448892 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:39:38.448892 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:39:38.448892 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:39:38.448892 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:39:38.448892 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:38.448892 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:38.448892 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:38.448892 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:39:38.448892 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:38.449135 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:38.449135 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:38.449135 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:39:38.449135 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:38.449135 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:39:38.449135 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:39:38.449135 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:38.449135 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:39:38.449135 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:38.449135 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:39:38.449135 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:39:38.449135 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:38.449135 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:39:38.449447 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:38.449447 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:39:38.449915 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:38.449915 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:39:38.449915 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:39:38.449915 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:39:38.449915 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:39:38.449915 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:39:38.449915 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:38.449915 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:39:38.449915 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:39:38.449915 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:38.449915 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:39:38.449915 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:39:38.449915 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:38.449915 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:38.449915 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:39:38.449915 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:38.450337 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:39:38.450337 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:39:38.450337 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:38.450337 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:38.450337 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:38.450337 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:39:38.450337 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:38.450337 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:38.450337 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:38.450337 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:39:38.450337 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:38.450337 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:39:38.450337 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:39:38.450337 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:38.450337 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:39:38.450337 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:38.450337 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:39:38.450337 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:39:38.450337 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:38.450818 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:39:38.450818 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:38.450818 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:39:42.962932 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:39:42.963840 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:39:42.964026 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:39:42.965430 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:39:43.000748 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:39:43.001214 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:43.002232 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:39:43.002232 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:43.002232 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:39:43.002232 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:39:43.002232 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:43.002232 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:39:43.002232 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:43.002232 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:39:43.003226 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.038 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:43.003226 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:39:43.003226 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:43.003226 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:39:43.003226 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:39:43.003226 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:39:43.003226 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:39:43.003226 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:43.003226 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:43.003226 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:43.003226 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:39:43.003226 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:43.003601 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:43.003601 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:43.003601 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:39:43.003601 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:43.003601 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:39:43.003601 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:39:43.003601 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:43.003601 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:39:43.003601 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:43.003601 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:39:43.003601 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:39:43.003601 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:43.003601 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:39:43.003923 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:43.003923 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:39:43.004684 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:43.004684 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:39:43.004684 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:39:43.004684 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:39:43.004684 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:39:43.004684 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:39:43.004684 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:43.004684 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:39:43.004684 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:39:43.004684 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:43.004684 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:39:43.004684 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:39:43.004684 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:43.004684 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:43.004684 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:39:43.004684 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:43.006454 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:39:43.006454 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:39:43.006454 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:43.006454 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:43.006454 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:43.006454 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:39:43.006454 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:43.006454 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:43.006454 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:43.006454 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:39:43.006454 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:43.006454 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:39:43.006454 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:39:43.006454 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:43.006454 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:39:43.006454 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:43.006454 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:39:43.006454 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:39:43.006454 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:43.007093 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:39:43.007093 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:43.007093 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:39:43.427661 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:39:43.428227 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:39:43.428463 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:39:43.429409 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:39:43.429746 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:43.430733 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:39:43.430733 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:43.430733 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:39:43.430733 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:39:43.430733 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:43.430733 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:39:43.430733 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:43.430733 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:39:43.432289 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:43.432289 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:39:43.432289 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:43.432289 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:39:43.432289 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:39:43.432289 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:39:43.432289 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:39:43.432289 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:43.432289 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:43.432289 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:43.432289 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:39:43.432289 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:43.432815 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:43.432815 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:43.432815 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:39:43.432815 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:43.432815 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:39:43.432815 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:39:43.432815 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:43.432815 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:39:43.432815 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:43.432815 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:39:43.432815 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:39:43.432815 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:43.432815 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:39:43.433299 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:43.433299 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:39:43.434095 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:43.434095 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:39:43.434095 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:39:43.434095 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:39:43.434095 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:39:43.434095 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:39:43.434095 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:43.434095 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:39:43.434095 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:39:43.434095 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:43.434095 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:39:43.434095 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:39:43.434095 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:43.434095 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:43.434095 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:39:43.434095 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:43.434789 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:39:43.434789 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:39:43.434789 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:43.434789 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:43.434789 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:43.434789 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:39:43.434789 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:43.434789 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:43.434789 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:43.434789 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:39:43.434789 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:43.434789 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:39:43.434789 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:39:43.434789 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:43.434789 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:39:43.434789 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:43.434789 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:39:43.434789 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:39:43.434789 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:43.435665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:39:43.435665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:43.435665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:39:43.435665 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:39:43.435665 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:39:43.435665 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:39:43.435665 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:39:43.435665 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:39:43.436072 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:43.436670 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:39:43.436670 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:43.436670 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:39:43.436670 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:39:43.436670 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:43.436670 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:39:43.436670 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:43.436670 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:39:43.437453 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:43.437453 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:39:43.437453 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:39:43.437453 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:39:43.437453 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:43.437453 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:39:43.437453 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:43.437453 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:39:43.437453 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:43.437453 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:39:43.437453 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:43.437453 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:39:43.437453 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:43.437453 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:39:43.437453 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:39:43.437875 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:43.437875 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:39:43.437875 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:43.437875 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:39:43.437875 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:39:43.437875 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:43.437875 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:39:43.437875 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:43.437875 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:39:43.438280 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:43.438280 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:39:43.438280 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:43.438280 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:39:43.438280 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:39:43.438280 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:39:43.438280 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:39:43.438280 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:39:43.438280 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:39:43.438280 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:39:43.438280 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:43.438280 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:43.438648 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:43.438648 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:39:43.438648 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:43.438648 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:43.438648 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:43.438648 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:39:43.438648 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:43.438648 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:39:43.438648 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:39:43.438648 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:43.438648 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:39:43.438648 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:43.438648 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:39:43.438648 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:39:43.439033 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:43.439033 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:39:43.439033 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:43.439033 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:39:43.439310 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:43.439310 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:39:43.439310 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:39:43.439310 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:39:43.439310 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:39:43.439310 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:39:43.439310 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:43.439310 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:39:43.439310 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:39:43.439310 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:43.439310 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:39:43.439310 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:39:43.439310 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:43.439310 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:43.439310 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:39:43.439310 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:43.439740 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:39:43.439740 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:39:43.439740 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:39:43.439740 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:39:43.439740 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:39:43.439740 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:43.439740 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:43.439740 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:43.439740 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:39:43.439740 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:43.439740 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:43.439740 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:43.439740 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:39:43.439740 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:43.439740 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:39:43.439740 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:39:43.439740 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:43.439740 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:39:43.440241 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:43.440241 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:39:43.440241 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:39:43.440241 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:43.440241 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:39:43.440241 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:43.440241 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:39:43.440241 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:39:43.440676 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:39:43.440910 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:39:43.441156 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:39:43.441713 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:39:43.442147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:43.442976 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:39:43.442976 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:43.442976 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:39:43.442976 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:39:43.442976 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:43.442976 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:39:43.442976 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:43.442976 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:39:43.444596 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:43.444596 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:39:43.444596 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:39:43.444596 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:39:43.444596 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:43.444596 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:39:43.444596 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:43.444596 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:39:43.444596 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:43.444596 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:39:43.444596 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:43.444596 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:39:43.444596 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:43.444596 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:39:43.444596 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:39:43.444996 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:43.444996 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:39:43.444996 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:43.444996 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:39:43.444996 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:39:43.444996 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:43.444996 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:39:43.444996 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:43.444996 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:39:43.446389 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:43.446389 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:39:43.446389 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:43.446389 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:39:43.446389 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:39:43.446389 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:39:43.446389 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:39:43.446389 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:39:43.446389 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:39:43.446389 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:43.446389 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:43.446711 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:43.446711 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:39:43.446711 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:43.446711 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:43.446711 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:43.446711 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:39:43.446711 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:43.446711 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:39:43.446711 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:39:43.446711 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:43.446711 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:39:43.446711 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:43.446711 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:39:43.446711 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:39:43.447086 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:43.447086 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:39:43.447086 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:43.447086 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:39:43.447086 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:39:43.448118 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:43.448118 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:39:43.448118 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:39:43.448118 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:39:43.448118 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:39:43.448118 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:39:43.448118 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:43.448118 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:39:43.448118 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:39:43.448118 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:43.448118 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:39:43.448118 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:39:43.448118 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:43.448118 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:43.448118 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:39:43.448118 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:43.448118 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:39:43.448614 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:39:43.448614 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:39:43.448614 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:39:43.448614 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:43.448614 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:43.448614 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:43.448614 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:39:43.448614 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:43.448614 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:43.448614 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:43.448614 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:39:43.448614 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:43.448614 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:39:43.448614 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:39:43.448614 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:43.448614 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:39:43.448614 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:43.448614 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:39:43.448614 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:39:43.449117 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:43.449117 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:39:43.449117 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:43.449117 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:39:43.449117 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:39:43.449117 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:39:43.449117 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:39:43.449337 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:39:43.449579 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:39:43.449761 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:43.450557 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:39:43.450557 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:43.450557 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:39:43.450557 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:39:43.450557 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:43.450557 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:39:43.450557 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:43.450557 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:39:43.451449 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:43.451449 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:39:43.451449 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:43.451449 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:39:43.451449 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:39:43.451449 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:39:43.451449 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:39:43.451449 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:43.451449 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:43.451449 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:43.451449 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:39:43.451449 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:43.451803 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:43.451803 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:43.451803 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:39:43.451803 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:43.451803 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:39:43.451803 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:39:43.451803 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:43.451803 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:39:43.451803 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:43.451803 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:39:43.451803 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:39:43.451803 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:43.451803 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:39:43.452127 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:43.452127 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:39:43.452505 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:43.452505 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:39:43.452505 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:39:43.452505 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:39:43.452505 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:39:43.452505 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:39:43.452505 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:43.452505 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:39:43.452505 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:39:43.452505 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:43.452505 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:39:43.452505 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:39:43.452505 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:43.452505 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:43.452505 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:39:43.452505 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:43.452940 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:39:43.452940 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:39:43.452940 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:43.452940 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:43.452940 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:43.452940 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:39:43.452940 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:43.452940 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:43.452940 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:43.452940 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:39:43.452940 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:43.452940 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:39:43.452940 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:39:43.452940 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:43.452940 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:39:43.452940 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:43.452940 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:39:43.452940 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:39:43.452940 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:43.453468 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:39:43.453468 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:43.453468 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:39:47.963791 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:39:47.964795 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:39:47.964795 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:39:47.966060 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:39:47.967905 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG dbcounter [-] [106123] Writing DB stats neutron:SELECT=304,neutron:UPDATE=1 {{(pid=106123) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} May 18 18:39:47.995452 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:39:47.995871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:47.996609 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:39:47.996609 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:47.996609 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:39:47.996609 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:39:47.996609 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:47.996609 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:39:47.996609 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:47.996609 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:39:47.997656 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.032 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:47.997656 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:39:47.997656 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:47.997656 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:39:47.997656 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:39:47.997656 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:39:47.997656 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:39:47.997656 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:47.997656 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:47.997656 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:47.997656 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:39:47.997656 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:47.998252 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:47.998252 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:47.998252 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:39:47.998252 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:47.998252 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:39:47.998252 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:39:47.998252 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:47.998252 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:39:47.998252 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:47.998252 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:39:47.998252 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:39:47.998252 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:47.998252 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:39:47.998875 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:47.998875 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:39:47.998875 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:47.998875 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:39:47.998875 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:39:47.998875 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:39:47.998875 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:39:47.998875 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:39:47.998875 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:47.998875 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:39:47.998875 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:39:47.998875 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:47.998875 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:39:47.998875 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:39:47.998875 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:47.999722 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:47.999722 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:39:47.999722 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:47.999722 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:39:47.999722 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:39:47.999722 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:47.999722 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:47.999722 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:47.999722 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:39:47.999722 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:47.999722 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:47.999722 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:47.999722 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:39:47.999722 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:47.999722 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:39:47.999722 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:39:47.999722 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:47.999722 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:39:47.999722 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:48.001224 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:39:48.001224 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:39:48.001224 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:48.001224 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:39:48.001224 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:48.001224 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:39:48.428231 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:39:48.428846 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:39:48.429010 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:39:48.429885 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:39:48.430183 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:48.431043 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:39:48.431043 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:48.431043 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:39:48.431043 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:39:48.431043 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:48.431043 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:39:48.431043 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:48.431043 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:39:48.432519 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:48.432519 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:39:48.432519 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:48.432519 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:39:48.432519 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:39:48.432519 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:39:48.432519 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:39:48.432519 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:48.432519 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:48.432519 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:48.432519 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:39:48.432519 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:48.434008 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:48.434008 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:48.434008 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:39:48.434008 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:48.434008 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:39:48.434008 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:39:48.434008 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:48.434008 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:39:48.434008 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:48.434008 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:39:48.434008 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:39:48.434008 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:48.434008 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:39:48.435100 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:48.435100 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:39:48.435100 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:48.435100 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:39:48.435100 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:39:48.435100 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:39:48.435100 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:39:48.435100 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:39:48.435100 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:48.435100 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:39:48.435100 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:39:48.435100 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:48.435100 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:39:48.435100 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:39:48.435100 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:48.436091 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:48.436091 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:39:48.436091 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:48.436091 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:39:48.436091 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:39:48.436091 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:48.436091 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:48.436091 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:48.436091 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:39:48.436091 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:48.436091 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:48.436091 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:48.436091 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:39:48.436091 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:48.436091 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:39:48.436091 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:39:48.436091 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:48.436091 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:39:48.436091 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:48.437734 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:39:48.437734 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:39:48.437734 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:48.437734 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:39:48.437734 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:48.437734 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:39:48.437734 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:39:48.437734 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:39:48.437734 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:39:48.437734 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:39:48.438425 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:39:48.438425 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:48.438425 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:39:48.438425 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:48.438425 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:39:48.438425 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:39:48.438425 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:48.438425 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:39:48.438425 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:48.438425 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:39:48.438425 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:48.438425 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:39:48.438425 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:39:48.438425 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:39:48.439230 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:48.439230 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:39:48.439230 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:48.439230 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:39:48.439230 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:48.439230 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:39:48.439230 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:48.439230 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:39:48.439230 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:48.439230 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:39:48.439230 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:39:48.439230 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:48.439230 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:39:48.439230 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:48.439230 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:39:48.439230 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:39:48.439230 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:48.439230 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:39:48.439787 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:48.439787 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:39:48.439848 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:48.439848 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:39:48.439848 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:48.439848 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:39:48.439848 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:39:48.439848 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:39:48.439848 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:39:48.439848 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:39:48.439848 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:39:48.439848 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:39:48.439848 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:48.439848 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:48.440151 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:48.440151 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:39:48.440151 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:48.440151 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:48.440151 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:48.440151 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:39:48.440151 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:48.440151 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:39:48.440151 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:39:48.440151 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:48.440151 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:39:48.440151 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:48.440151 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:39:48.440151 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:39:48.440520 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:48.440520 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:39:48.440520 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:48.440520 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:39:48.441195 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:48.441195 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:39:48.441195 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:39:48.441195 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:39:48.441195 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:39:48.441195 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:39:48.441195 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:48.441195 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:39:48.441195 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:39:48.441195 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:48.441195 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:39:48.441195 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:39:48.441195 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:48.441195 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:48.441195 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:39:48.441195 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:48.441776 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:39:48.441776 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:39:48.441776 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:39:48.441776 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:39:48.441776 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:39:48.441776 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:48.441776 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:48.441776 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:48.441776 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:39:48.441776 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:48.441776 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:48.441776 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:48.441776 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:39:48.441776 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:48.441776 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:39:48.441776 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:39:48.441776 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:48.441776 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:39:48.442226 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:48.442226 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:39:48.442226 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:39:48.442226 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:48.442226 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:39:48.442226 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:48.442226 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:39:48.442226 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:39:48.442226 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:39:48.442226 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:39:48.442519 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:39:48.442519 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:39:48.442519 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:48.443010 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:39:48.443010 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:48.443010 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:39:48.443010 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:39:48.443010 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:48.443010 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:39:48.443010 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:48.443010 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:39:48.443649 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:48.443649 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:39:48.443649 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:39:48.443649 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:39:48.443649 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:48.443649 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:39:48.443649 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:48.443649 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:39:48.443649 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:48.443649 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:39:48.443649 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:48.443649 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:39:48.443649 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:48.443649 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:39:48.443649 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:39:48.444616 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:48.444616 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:39:48.444616 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:48.444616 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:39:48.444616 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:39:48.444616 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:48.444616 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:39:48.444616 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:48.444616 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:39:48.444616 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:48.444616 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:39:48.444616 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:48.444616 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:39:48.444616 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:39:48.445492 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:39:48.445492 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:39:48.445492 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:39:48.445492 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:39:48.445492 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:48.445492 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:48.445492 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:48.445492 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:39:48.445492 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:48.445492 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:48.445492 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:48.445492 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:39:48.445492 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:48.446237 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:39:48.446237 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:39:48.446237 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:48.446237 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:39:48.446237 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:48.446237 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:39:48.446237 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:39:48.446237 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:48.446237 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:39:48.446237 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:48.446237 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:39:48.446237 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:48.446560 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:39:48.446560 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:39:48.446560 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:39:48.446560 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:39:48.446560 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:39:48.446560 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:48.446560 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:39:48.446560 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:39:48.446560 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:48.446560 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:39:48.446560 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:39:48.446560 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:48.446560 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:48.446560 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:39:48.446560 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:48.446560 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:39:48.446560 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:39:48.446560 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:39:48.446560 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:39:48.447012 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:48.447012 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:48.447012 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:48.447012 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:39:48.447012 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:48.447012 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:48.447012 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:48.447012 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:39:48.447012 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:48.447012 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:39:48.447012 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:39:48.447012 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:48.447012 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:39:48.447012 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:48.447012 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:39:48.447012 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:39:48.447012 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:48.447012 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:39:48.447012 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:48.447012 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:39:48.448423 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:39:48.448814 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:39:48.449008 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:39:48.449258 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:39:48.449723 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:39:48.450092 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:39:48.450287 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:48.451083 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:39:48.451083 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:48.451083 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:39:48.451083 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:39:48.451083 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:48.451083 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:39:48.451083 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:48.451083 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:39:48.452569 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:48.452569 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:39:48.452569 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:48.452569 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:39:48.452569 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:39:48.452569 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:39:48.452569 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:39:48.452569 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:48.452569 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:48.452569 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:48.452569 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:39:48.452569 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:48.453058 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:48.453058 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:48.453058 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:39:48.453058 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:48.453058 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:39:48.453058 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:39:48.453058 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:48.453058 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:39:48.453058 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:48.453058 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:39:48.453058 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:39:48.453058 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:48.453058 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:39:48.453514 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:48.453514 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:39:48.454282 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:48.454282 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:39:48.454282 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:39:48.454282 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:39:48.454282 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:39:48.454282 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:39:48.454282 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:48.454282 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:39:48.454282 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:39:48.454282 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:48.454282 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:39:48.454282 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:39:48.454282 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:48.454282 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:48.454282 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:39:48.454282 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:48.454898 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:39:48.454898 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:39:48.454898 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:48.454898 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:48.454898 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:48.454898 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:39:48.454898 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:48.454898 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:48.454898 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:48.454898 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:39:48.454898 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:48.454898 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:39:48.454898 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:39:48.454898 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:48.454898 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:39:48.454898 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:48.454898 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:39:48.454898 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:39:48.454898 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:48.455592 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:39:48.455592 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:48.455592 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:39:52.964817 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:39:52.965830 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:39:52.965830 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:39:52.967211 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:39:52.996842 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:39:52.997247 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:52.998174 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:39:52.998174 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:52.998174 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:39:52.998174 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:39:52.998174 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:52.998174 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:39:52.998174 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:52.998174 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:39:52.998984 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.032 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:52.998984 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:39:52.998984 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:52.998984 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:39:52.998984 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:39:52.998984 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:39:52.998984 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:39:52.998984 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:52.998984 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:52.998984 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:52.998984 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:39:52.998984 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:52.999565 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:52.999565 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:52.999565 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:39:52.999565 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:52.999565 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:39:52.999565 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:39:52.999565 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:52.999565 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:39:52.999565 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:52.999565 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:39:52.999565 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:39:52.999565 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:52.999565 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:39:53.000528 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:53.000528 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:39:53.000528 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:53.000528 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:39:53.000528 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:39:53.000528 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:39:53.000528 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:39:53.000528 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:39:53.000528 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:53.000528 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:39:53.000528 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:39:53.000528 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:53.000528 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:39:53.000528 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:39:53.000528 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:53.001443 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:53.001443 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:39:53.001443 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:53.001443 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:39:53.001443 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:39:53.001443 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:53.001443 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:53.001443 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:53.001443 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:39:53.001443 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:53.001443 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:53.001443 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:53.001443 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:39:53.001443 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:53.001443 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:39:53.001443 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:39:53.001443 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:53.001443 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:39:53.001443 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:53.002216 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:39:53.002216 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:39:53.002216 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:53.002216 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:39:53.002216 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:53.002216 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:39:53.429143 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:39:53.429784 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:39:53.429992 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:39:53.431074 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:39:53.431516 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:53.432576 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:39:53.432576 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:53.432576 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:39:53.432576 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:39:53.432576 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:53.432576 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:39:53.432576 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:53.432576 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:39:53.434295 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:53.434295 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:39:53.434295 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:53.434295 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:39:53.434295 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:39:53.434295 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:39:53.434295 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:39:53.434295 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:53.434295 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:53.434295 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:53.434295 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:39:53.434295 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:53.435116 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:53.435116 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:53.435116 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:39:53.435116 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:53.435116 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:39:53.435116 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:39:53.435116 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:53.435116 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:39:53.435116 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:53.435116 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:39:53.435116 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:39:53.435116 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:53.435116 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:39:53.435460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:53.435460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:39:53.435460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:53.435460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:39:53.435460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:39:53.435460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:39:53.435460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:39:53.435460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:39:53.435460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:53.435460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:39:53.435460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:39:53.435460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:53.435460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:39:53.435460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:39:53.435460 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:53.435838 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:53.435838 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:39:53.435838 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:53.435838 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:39:53.435838 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:39:53.435838 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:53.435838 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:53.435838 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:53.435838 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:39:53.435838 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:53.435838 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:53.435838 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:53.435838 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:39:53.435838 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:53.435838 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:39:53.435838 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:39:53.435838 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:53.435838 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:39:53.435838 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:53.436309 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:39:53.436309 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:39:53.436309 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:53.436309 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:39:53.436309 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:53.436309 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:39:53.436309 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:39:53.436309 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:39:53.436309 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:39:53.436309 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:39:53.436840 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:39:53.437156 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:53.437927 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:39:53.437927 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:53.437927 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:39:53.437927 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:39:53.437927 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:53.437927 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:39:53.437927 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:53.437927 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:39:53.438678 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:53.438678 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:39:53.438678 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:39:53.438678 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:39:53.438678 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:53.438678 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:39:53.438678 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:53.438678 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:39:53.438678 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:53.438678 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:39:53.438678 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:53.438678 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:39:53.438678 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:53.438678 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:39:53.438678 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:39:53.439345 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:53.439345 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:39:53.439345 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:53.439345 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:39:53.439345 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:39:53.439345 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:53.439345 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:39:53.439345 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:53.439345 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:39:53.439587 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:53.439587 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:39:53.439587 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:53.439587 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:39:53.439587 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:39:53.439587 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:39:53.439587 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:39:53.439587 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:39:53.439587 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:39:53.439587 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:39:53.439587 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:53.439587 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:53.439896 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:53.439896 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:39:53.439896 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:53.439896 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:53.439896 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:53.439896 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:39:53.439896 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:53.439896 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:39:53.439896 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:39:53.439896 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:53.439896 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:39:53.439896 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:53.439896 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:39:53.439896 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:39:53.440225 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:53.440225 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:39:53.440225 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:53.440225 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:39:53.440544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:53.440544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:39:53.440544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:39:53.440544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:39:53.440544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:39:53.440544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:39:53.440544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:53.440544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:39:53.440544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:39:53.440544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:53.440544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:39:53.440544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:39:53.440544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:53.440544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:53.440544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:39:53.440544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:53.440931 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:39:53.440931 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:39:53.440931 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:39:53.440931 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:39:53.440931 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:39:53.440931 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:53.440931 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:53.440931 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:53.440931 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:39:53.440931 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:53.440931 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:53.440931 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:53.440931 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:39:53.440931 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:53.440931 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:39:53.440931 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:39:53.440931 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:53.440931 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:39:53.441406 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:53.441406 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:39:53.441406 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:39:53.441406 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:53.441406 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:39:53.441406 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:53.441406 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:39:53.441794 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:39:53.442072 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:39:53.442171 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:39:53.442303 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:39:53.442769 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:39:53.443121 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:53.443812 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:39:53.443812 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:53.443812 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:39:53.443812 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:39:53.443812 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:53.443812 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:39:53.443812 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:53.443812 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:39:53.444545 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:53.444545 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:39:53.444545 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:39:53.444545 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:39:53.444545 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:53.444545 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:39:53.444545 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:53.444545 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:39:53.444545 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:53.444545 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:39:53.444545 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:53.444545 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:39:53.444545 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:53.444545 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:39:53.444545 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:39:53.445302 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:53.445302 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:39:53.445302 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:53.445302 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:39:53.445302 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:39:53.445302 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:53.445302 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:39:53.445302 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:53.445302 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:39:53.445878 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:53.445878 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:39:53.445878 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:53.445878 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:39:53.445878 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:39:53.445878 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:39:53.445878 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:39:53.445878 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:39:53.445878 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:39:53.445878 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:53.445878 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:53.446520 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:53.446520 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:39:53.446520 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:53.446520 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:53.446520 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:53.446520 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:39:53.446520 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:53.446520 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:39:53.446520 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:39:53.446520 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:53.446520 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:39:53.446520 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:53.446520 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:39:53.446520 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:39:53.447026 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:53.447026 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:39:53.447026 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:53.447026 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:39:53.447026 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:53.447026 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:39:53.447026 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:39:53.447026 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:39:53.447026 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:39:53.447026 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:39:53.447026 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:53.447026 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:39:53.447026 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:39:53.447026 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:53.447410 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:39:53.447410 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:39:53.447410 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:53.447410 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:53.447410 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:39:53.447410 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:53.447410 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:39:53.447410 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:39:53.447410 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:39:53.447410 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:39:53.447410 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:53.447410 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:53.447410 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:53.447410 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:39:53.447410 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:53.447410 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:53.447410 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:53.447410 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:39:53.447410 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:53.447929 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:39:53.447929 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:39:53.447929 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:53.447929 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:39:53.447929 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:53.447929 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:39:53.447929 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:39:53.447929 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:53.447929 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:39:53.447929 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:53.447929 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:39:53.448976 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:39:53.449156 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:39:53.449257 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:39:53.449453 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:39:53.449789 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:39:53.449997 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:39:53.450134 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:53.450621 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:39:53.450621 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:53.450621 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:39:53.450621 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:39:53.450621 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:53.450621 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:39:53.450621 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:53.450621 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:39:53.451792 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:53.451792 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:39:53.451792 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:53.451792 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:39:53.451792 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:39:53.451792 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:39:53.451792 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:39:53.451792 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:53.451792 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:53.451792 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:53.451792 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:39:53.451792 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:53.452494 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:53.452494 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:53.452494 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:39:53.452494 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:53.452494 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:39:53.452494 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:39:53.452494 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:53.452494 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:39:53.452494 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:53.452494 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:39:53.452494 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:39:53.452494 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:53.452494 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:39:53.453191 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:53.453191 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:39:53.453191 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:53.453191 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:39:53.453191 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:39:53.453191 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:39:53.453191 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:39:53.453191 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:39:53.453191 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:53.453191 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:39:53.453191 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:39:53.453191 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:53.453191 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:39:53.453191 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:39:53.453191 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:53.454038 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:53.454038 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:39:53.454038 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:53.454038 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:39:53.454038 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:39:53.454038 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:53.454038 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:53.454038 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:53.454038 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:39:53.454038 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:53.454038 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:53.454038 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:53.454038 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:39:53.454038 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:53.454038 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:39:53.454038 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:39:53.454038 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:53.454038 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:39:53.454038 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:53.454767 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:39:53.454767 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:39:53.454767 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:53.454767 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:39:53.454767 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:53.454767 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:39:57.965402 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:39:57.966428 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:39:57.966428 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:39:57.966703 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:39:57.996958 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:39:57.997346 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:57.998095 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:39:57.998095 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:57.998095 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:39:57.998095 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:39:57.998095 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:57.998095 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:39:57.998095 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:57.998095 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:39:57.999012 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.032 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:57.999012 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:39:57.999012 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:57.999012 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:39:57.999012 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:39:57.999012 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:39:57.999012 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:39:57.999012 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:57.999012 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:57.999012 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:57.999012 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:39:57.999012 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:57.999693 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:57.999693 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:57.999693 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:39:57.999693 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:57.999693 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:39:57.999693 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:39:57.999693 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:57.999693 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:39:57.999693 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:57.999693 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:39:57.999693 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:39:57.999693 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:57.999693 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:39:58.000930 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:58.000930 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:39:58.000930 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:58.000930 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:39:58.000930 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:39:58.000930 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:39:58.000930 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:39:58.000930 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:39:58.000930 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:58.000930 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:39:58.000930 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:39:58.000930 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:58.000930 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:39:58.000930 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:39:58.000930 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:58.001648 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:58.001648 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:39:58.001648 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:58.001648 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:39:58.001648 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:39:58.001648 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:58.001648 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:58.001648 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:58.001648 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:39:58.001648 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:58.001648 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:58.001648 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:58.001648 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:39:58.001648 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:58.001648 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:39:58.001648 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:39:58.001648 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:58.001648 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:39:58.001648 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:58.003249 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:39:58.003249 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:39:58.003249 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:58.003249 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:39:58.003249 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:58.003249 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:39:58.430001 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:39:58.430600 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:39:58.430777 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:39:58.431922 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:39:58.432251 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:58.433148 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:39:58.433148 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:58.433148 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:39:58.433148 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:39:58.433148 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:58.433148 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:39:58.433148 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:58.433148 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:39:58.434770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:58.434770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:39:58.434770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:58.434770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:39:58.434770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:39:58.434770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:39:58.434770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:39:58.434770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:58.434770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:58.434770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:58.434770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:39:58.434770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:58.435372 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:58.435372 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:58.435372 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:39:58.435372 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:58.435372 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:39:58.435372 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:39:58.435372 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:58.435372 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:39:58.435372 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:58.435372 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:39:58.435372 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:39:58.435372 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:58.435372 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:39:58.435880 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:58.435880 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:39:58.436573 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:58.436573 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:39:58.436573 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:39:58.436573 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:39:58.436573 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:39:58.436573 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:39:58.436573 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:58.436573 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:39:58.436573 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:39:58.436573 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:58.436573 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:39:58.436573 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:39:58.436573 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:58.436573 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:58.436573 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:39:58.436573 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:58.437315 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:39:58.437315 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:39:58.437315 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:58.437315 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:58.437315 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:58.437315 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:39:58.437315 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:58.437315 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:58.437315 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:58.437315 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:39:58.437315 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:58.437315 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:39:58.437315 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:39:58.437315 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:58.437315 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:39:58.437315 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:58.437315 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:39:58.437315 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:39:58.437315 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:58.438145 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:39:58.438145 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:58.438145 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:39:58.438145 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:39:58.438145 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:39:58.438145 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:39:58.438145 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:39:58.438529 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:39:58.438529 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:58.439259 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:39:58.439259 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:58.439259 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:39:58.439259 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:39:58.439259 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:58.439259 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:39:58.439259 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:58.439259 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:39:58.440225 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:58.440225 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:39:58.440225 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:39:58.440225 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:39:58.440225 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:58.440225 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:39:58.440225 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:58.440225 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:39:58.440225 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:58.440225 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:39:58.440225 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:58.440225 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:39:58.440225 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:58.440225 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:39:58.440225 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:39:58.440725 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:58.440725 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:39:58.440725 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:58.440725 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:39:58.440725 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:39:58.440725 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:58.440725 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:39:58.440725 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:58.440725 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:39:58.441011 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:58.441011 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:39:58.441011 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:58.441011 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:39:58.441011 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:39:58.441011 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:39:58.441011 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:39:58.441011 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:39:58.441011 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:39:58.441011 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:39:58.441011 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:58.441011 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:58.441447 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:58.441447 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:39:58.441447 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:58.441447 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:58.441447 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:58.441447 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:39:58.441447 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:58.441447 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:39:58.441447 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:39:58.441447 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:58.441447 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:39:58.441447 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:58.441447 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:39:58.441447 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:39:58.442121 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:58.442121 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:39:58.442121 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:58.442121 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:39:58.442121 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:58.442121 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:39:58.442121 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:39:58.442121 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:39:58.442121 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:39:58.442121 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:39:58.442121 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:58.442121 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:39:58.442121 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:39:58.442121 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:58.442544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:39:58.442544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:39:58.442544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:58.442544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:58.442544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:39:58.442544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:58.442544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:39:58.442544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:39:58.442544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:39:58.442544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:39:58.442544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:39:58.442544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:58.442544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:58.442544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:58.442544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:39:58.442544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:58.442544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:58.442544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:58.442544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:39:58.443086 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:58.443086 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:39:58.443086 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:39:58.443086 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:58.443086 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:39:58.443086 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:58.443086 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:39:58.443086 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:39:58.443086 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:58.443086 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:39:58.443086 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:58.443086 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:39:58.443086 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:39:58.443086 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:39:58.443563 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:39:58.443563 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:39:58.443563 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:39:58.443563 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:58.444246 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:39:58.444246 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:58.444246 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:39:58.444246 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:39:58.444246 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:58.444246 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:39:58.444246 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:58.444246 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:39:58.444983 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:58.444983 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:39:58.444983 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:39:58.444983 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:39:58.444983 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:58.444983 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:39:58.444983 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:58.444983 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:39:58.444983 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:58.444983 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:39:58.444983 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:58.444983 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:39:58.444983 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:58.444983 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:39:58.444983 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:39:58.445423 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:58.445423 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:39:58.445423 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:58.445423 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:39:58.445423 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:39:58.445423 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:58.445423 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:39:58.445423 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:58.445423 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:39:58.445851 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:58.445851 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:39:58.445851 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:58.445851 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:39:58.445851 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:39:58.445851 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:39:58.445851 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:39:58.445851 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:39:58.445851 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:39:58.445851 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:58.445851 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:58.446186 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:58.446186 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:39:58.446186 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:58.446186 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:58.446186 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:58.446186 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:39:58.446186 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:58.446186 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:39:58.446186 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:39:58.446186 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:58.446186 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:39:58.446186 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:58.446186 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:39:58.446186 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:39:58.446604 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:58.446604 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:39:58.446604 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:58.446604 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:39:58.446908 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:58.446908 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:39:58.446908 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:39:58.446908 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:39:58.446908 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:39:58.446908 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:39:58.446908 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:58.446908 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:39:58.446908 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:39:58.446908 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:58.446908 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:39:58.446908 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:39:58.446908 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:58.446908 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:58.446908 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:39:58.446908 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:58.446908 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:39:58.447456 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:39:58.447456 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:39:58.447456 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:39:58.447456 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:58.447456 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:58.447456 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:58.447456 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:39:58.447456 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:58.447456 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:58.447456 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:58.447456 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:39:58.447456 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:58.447456 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:39:58.447456 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:39:58.447456 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:58.447456 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:39:58.447456 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:58.447456 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:39:58.447456 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:39:58.447954 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:58.447954 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:39:58.447954 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:58.447954 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:39:58.449345 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:39:58.449623 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:39:58.449739 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:39:58.449891 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:39:58.450202 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:39:58.450438 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:39:58.450554 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:58.451016 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:39:58.451016 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:58.451016 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:39:58.451016 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:39:58.451016 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:58.451016 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:39:58.451016 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:58.451016 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:39:58.451886 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:58.451886 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:39:58.451886 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:58.451886 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:39:58.451886 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:39:58.451886 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:39:58.451886 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:39:58.451886 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:58.451886 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:58.451886 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:58.451886 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:39:58.451886 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:58.452262 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:39:58.452262 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:58.452262 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:39:58.452262 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:58.452262 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:39:58.452262 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:39:58.452262 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:58.452262 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:39:58.452262 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:58.452262 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:39:58.452262 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:39:58.452262 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:58.452262 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:39:58.452657 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:58.452657 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:39:58.452890 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:58.452890 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:39:58.452890 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:39:58.452890 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:39:58.452890 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:39:58.452890 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:39:58.452890 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:58.452890 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:39:58.452890 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:39:58.452890 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:58.452890 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:39:58.452890 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:39:58.452890 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:39:58.452890 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:39:58.452890 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:39:58.452890 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:39:58.453478 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:39:58.453478 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:39:58.453478 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:58.453478 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:58.453478 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:39:58.453478 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:39:58.453478 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:39:58.453478 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:39:58.453478 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:39:58.453478 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:39:58.453478 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:39:58.453478 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:39:58.453478 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:39:58.453478 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:39:58.453478 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:39:58.453478 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:39:58.453478 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:39:58.453478 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:39:58.453478 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:39:58.454015 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:39:58.454015 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:39:58.454015 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:40:02.965997 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:40:02.967035 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:40:02.967035 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:40:02.968013 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:40:03.001645 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:40:03.002037 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:03.002966 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:40:03.002966 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:03.002966 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:40:03.002966 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:40:03.002966 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:03.002966 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:40:03.002966 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:03.002966 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:40:03.003795 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.036 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:03.003795 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:40:03.003795 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:03.003795 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:40:03.003795 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:40:03.003795 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:40:03.003795 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:40:03.003795 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:03.003795 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:03.003795 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:03.003795 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:40:03.003795 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:03.004382 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:03.004382 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:03.004382 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:40:03.004382 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:03.004382 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:40:03.004382 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:40:03.004382 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:03.004382 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:40:03.004382 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:03.004382 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:40:03.004382 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:40:03.004382 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:03.004382 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:40:03.004989 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:03.004989 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:40:03.004989 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:03.004989 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:40:03.004989 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:40:03.004989 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:40:03.004989 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:40:03.004989 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:40:03.004989 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:03.004989 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:40:03.004989 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:40:03.004989 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:03.004989 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:40:03.004989 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:40:03.004989 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:03.005724 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:03.005724 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:40:03.005724 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:03.005724 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:40:03.005724 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:40:03.005724 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:03.005724 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:03.005724 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:03.005724 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:40:03.005724 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:03.005724 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:03.005724 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:03.005724 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:40:03.005724 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:03.005724 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:40:03.005724 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:40:03.005724 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:03.005724 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:40:03.005724 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:03.006528 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:40:03.006528 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:40:03.006528 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:03.006528 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:40:03.006528 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:03.006528 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:40:03.430804 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:40:03.431181 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:40:03.431386 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:40:03.432562 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:40:03.432903 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:03.433878 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:40:03.433878 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:03.433878 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:40:03.433878 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:40:03.433878 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:03.433878 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:40:03.433878 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:03.433878 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:40:03.435594 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:03.435594 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:40:03.435594 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:03.435594 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:40:03.435594 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:40:03.435594 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:40:03.435594 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:40:03.435594 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:03.435594 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:03.435594 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:03.435594 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:40:03.435594 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:03.436348 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:03.436348 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:03.436348 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:40:03.436348 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:03.436348 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:40:03.436348 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:40:03.436348 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:03.436348 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:40:03.436348 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:03.436348 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:40:03.436348 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:40:03.436348 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:03.436348 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:40:03.437217 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:03.437217 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:40:03.437417 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:03.437417 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:40:03.437417 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:40:03.437417 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:40:03.437417 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:40:03.437417 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:40:03.437417 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:03.437417 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:40:03.437417 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:40:03.437417 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:03.437417 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:40:03.437417 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:40:03.437417 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:03.437417 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:03.437417 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:40:03.437417 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:03.438433 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:40:03.438433 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:40:03.438433 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:03.438433 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:03.438433 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:03.438433 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:40:03.438433 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:03.438433 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:03.438433 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:03.438433 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:40:03.438433 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:03.438433 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:40:03.438433 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:40:03.438433 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:03.438433 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:40:03.438433 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:03.438433 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:40:03.438433 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:40:03.438433 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:03.439597 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:40:03.439597 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:03.439597 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:40:03.439597 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:40:03.439597 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:40:03.439597 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:40:03.439597 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:40:03.439597 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:40:03.440302 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:03.440302 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:40:03.440302 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:03.440302 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:40:03.440302 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:40:03.440302 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:03.440302 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:40:03.440302 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:03.440302 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:40:03.440920 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:03.440920 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:40:03.440920 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:40:03.440920 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:40:03.440920 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:03.440920 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:40:03.440920 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:03.440920 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:40:03.440920 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:03.440920 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:40:03.440920 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:03.440920 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:40:03.440920 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:03.440920 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:40:03.440920 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:40:03.442006 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:03.442006 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:40:03.442006 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:03.442006 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:40:03.442006 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:40:03.442006 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:03.442006 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:40:03.442006 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:03.442006 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:40:03.442006 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:03.442006 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:40:03.442006 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:03.442006 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:40:03.442006 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:40:03.442555 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:40:03.442555 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:40:03.442555 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:40:03.442555 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:40:03.442555 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:40:03.442555 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:03.442555 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:03.442555 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:03.442555 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:40:03.442555 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:03.442555 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:03.442555 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:03.442555 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:40:03.443025 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:03.443025 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:40:03.443025 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:40:03.443025 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:03.443025 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:40:03.443025 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:03.443025 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:40:03.443025 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:40:03.443025 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:03.443025 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:40:03.443025 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:03.443025 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:40:03.443468 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:03.443468 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:40:03.443468 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:40:03.443468 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:40:03.443468 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:40:03.443468 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:40:03.443468 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:03.443468 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:40:03.443468 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:40:03.443468 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:03.443468 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:40:03.443468 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:40:03.443468 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:03.443468 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:03.443468 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:40:03.443468 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:03.443998 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:40:03.443998 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:40:03.443998 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:40:03.443998 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:40:03.443998 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:40:03.443998 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:03.443998 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:03.443998 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:03.443998 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:40:03.443998 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:03.443998 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:03.443998 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:03.443998 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:40:03.443998 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:03.443998 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:40:03.443998 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:40:03.443998 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:03.443998 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:40:03.444716 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:03.444716 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:40:03.444716 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:40:03.444716 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:03.444716 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:40:03.444716 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:03.444716 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:40:03.444716 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:40:03.444716 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:40:03.444716 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:40:03.445137 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:40:03.445137 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:40:03.445137 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:03.445137 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:40:03.445137 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:03.445137 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:40:03.445137 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:40:03.445137 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:03.445137 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:40:03.445137 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:03.445137 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:40:03.445548 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:03.445548 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:40:03.445548 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:40:03.445548 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:40:03.445548 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:03.445548 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:40:03.445548 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:03.445548 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:40:03.445548 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:03.445548 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:40:03.445548 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:03.445548 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:40:03.445548 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:03.445548 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:40:03.445548 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:40:03.446065 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:03.446065 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:40:03.446065 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:03.446065 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:40:03.446065 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:40:03.446065 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:03.446065 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:40:03.446065 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:03.446065 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:40:03.446611 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:03.446611 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:40:03.446611 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:03.446611 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:40:03.446611 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:40:03.446611 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:40:03.446611 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:40:03.446611 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:40:03.446611 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:40:03.446611 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:03.446611 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:03.447039 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:03.447039 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:40:03.447039 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:03.447039 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:03.447039 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:03.447039 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:40:03.447039 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:03.447039 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:40:03.447039 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:40:03.447039 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:03.447039 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:40:03.447039 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:03.447039 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:40:03.447039 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:40:03.447544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:03.447544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:40:03.447544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:03.447544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:40:03.447900 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:03.447900 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:40:03.447900 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:40:03.447900 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:40:03.447900 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:40:03.447900 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:40:03.447900 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:03.447900 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:40:03.447900 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:40:03.447900 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:03.447900 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:40:03.447900 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:40:03.447900 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:03.447900 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:03.447900 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:40:03.447900 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:03.447900 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:40:03.448560 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:40:03.448560 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:40:03.448560 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:40:03.448560 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:03.448560 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:03.448560 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:03.448560 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:40:03.448560 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:03.448560 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:03.448560 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:03.448560 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:40:03.448560 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:03.448560 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:40:03.448560 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:40:03.448560 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:03.448560 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:40:03.448560 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:03.448560 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:40:03.448560 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:40:03.449246 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:03.449246 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:40:03.449246 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:03.449246 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:40:03.449645 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:40:03.449983 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:40:03.450160 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:40:03.450427 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:40:03.450781 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:40:03.451017 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:40:03.451176 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:03.451996 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:40:03.451996 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:03.451996 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:40:03.451996 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:40:03.451996 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:03.451996 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:40:03.451996 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:03.451996 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:40:03.453373 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:03.453373 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:40:03.453373 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:03.453373 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:40:03.453373 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:40:03.453373 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:40:03.453373 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:40:03.453373 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:03.453373 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:03.453373 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:03.453373 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:40:03.453373 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:03.453697 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:03.453697 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:03.453697 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:40:03.453697 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:03.453697 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:40:03.453697 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:40:03.453697 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:03.453697 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:40:03.453697 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:03.453697 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:40:03.453697 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:40:03.453697 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:03.453697 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:40:03.454015 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:03.454015 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:40:03.454976 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:03.454976 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:40:03.454976 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:40:03.454976 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:40:03.454976 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:40:03.454976 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:40:03.454976 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:03.454976 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:40:03.454976 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:40:03.454976 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:03.454976 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:40:03.454976 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:40:03.454976 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:03.454976 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:03.454976 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:40:03.454976 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:03.455408 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:40:03.455408 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:40:03.455408 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:03.455408 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:03.455408 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:03.455408 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:40:03.455408 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:03.455408 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:03.455408 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:03.455408 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:40:03.455408 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:03.455408 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:40:03.455408 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:40:03.455408 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:03.455408 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:40:03.455408 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:03.455408 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:40:03.455408 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:40:03.455408 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:03.455867 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:40:03.455867 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:03.455867 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:40:07.966767 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:40:07.967593 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:40:07.967830 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:40:07.969124 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:40:08.000375 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:40:08.000823 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:08.001844 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:40:08.001844 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:08.001844 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:40:08.001844 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:40:08.001844 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:08.001844 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:40:08.001844 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:08.001844 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:40:08.002614 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.034 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:08.002614 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:40:08.002614 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:08.002614 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:40:08.002614 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:40:08.002614 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:40:08.002614 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:40:08.002614 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:08.002614 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:08.002614 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:08.002614 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:40:08.002614 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:08.003068 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:08.003068 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:08.003068 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:40:08.003068 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:08.003068 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:40:08.003068 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:40:08.003068 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:08.003068 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:40:08.003068 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:08.003068 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:40:08.003068 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:40:08.003068 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:08.003068 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:40:08.003434 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:08.003434 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:40:08.003498 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:08.003498 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:40:08.003498 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:40:08.003498 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:40:08.003498 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:40:08.003498 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:40:08.003498 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:08.003498 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:40:08.003498 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:40:08.003498 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:08.003498 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:40:08.003498 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:40:08.003498 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:08.003498 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:08.003498 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:40:08.003498 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:08.003923 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:40:08.003923 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:40:08.003923 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:08.003923 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:08.003923 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:08.003923 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:40:08.003923 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:08.003923 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:08.003923 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:08.003923 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:40:08.003923 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:08.003923 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:40:08.003923 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:40:08.003923 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:08.003923 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:40:08.003923 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:08.003923 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:40:08.003923 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:40:08.003923 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:08.004427 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:40:08.004427 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:08.004427 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:40:08.431464 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:40:08.432066 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:40:08.432259 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:40:08.433139 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:40:08.433506 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:08.434349 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:40:08.434349 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:08.434349 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:40:08.434349 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:40:08.434349 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:08.434349 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:40:08.434349 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:08.434349 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:40:08.435926 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:08.435926 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:40:08.435926 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:08.435926 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:40:08.435926 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:40:08.435926 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:40:08.435926 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:40:08.435926 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:08.435926 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:08.435926 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:08.435926 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:40:08.435926 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:08.436616 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:08.436616 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:08.436616 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:40:08.436616 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:08.436616 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:40:08.436616 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:40:08.436616 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:08.436616 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:40:08.436616 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:08.436616 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:40:08.436616 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:40:08.436616 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:08.436616 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:40:08.437215 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:08.437215 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:40:08.437871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:08.437871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:40:08.437871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:40:08.437871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:40:08.437871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:40:08.437871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:40:08.437871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:08.437871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:40:08.437871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:40:08.437871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:08.437871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:40:08.437871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:40:08.437871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:08.437871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:08.437871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:40:08.437871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:08.439574 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:40:08.439574 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:40:08.439574 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:08.439574 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:08.439574 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:08.439574 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:40:08.439574 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:08.439574 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:08.439574 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:08.439574 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:40:08.439574 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:08.439574 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:40:08.439574 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:40:08.439574 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:08.439574 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:40:08.439574 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:08.439574 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:40:08.439574 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:40:08.439574 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:08.441640 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:40:08.441640 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:08.441640 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:40:08.441640 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:40:08.441640 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:40:08.441640 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:40:08.441640 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:40:08.441640 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:40:08.442091 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:08.442091 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:40:08.442091 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:08.442091 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:40:08.442091 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:40:08.442091 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:08.442091 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:40:08.442091 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:08.442091 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:40:08.442091 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:08.442091 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:40:08.442091 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:40:08.442091 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:40:08.442091 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:08.442091 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:40:08.442091 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:08.442091 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:40:08.443394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:08.443394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:40:08.443394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:08.443394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:40:08.443394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:08.443394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:40:08.443394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:40:08.443394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:08.443394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:40:08.443394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:08.443394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:40:08.443394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:40:08.443394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:08.443394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:40:08.443394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:08.443394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:40:08.443716 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:08.443716 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:40:08.443716 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:08.443716 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:40:08.443716 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:40:08.443716 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:40:08.443716 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:40:08.443716 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:40:08.443716 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:40:08.443716 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:40:08.443716 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:08.443716 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:08.443921 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:08.443921 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:40:08.443921 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:08.443921 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:08.443921 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:08.443921 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:40:08.443921 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:08.443921 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:40:08.443921 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:40:08.443921 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:08.443921 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:40:08.443921 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:08.443921 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:40:08.443921 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:40:08.444166 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:08.444166 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:40:08.444166 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:08.444166 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:40:08.444166 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:08.444166 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:40:08.444166 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:40:08.444166 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:40:08.444166 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:40:08.444166 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:40:08.444166 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:08.444166 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:40:08.444166 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:40:08.444166 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:08.444423 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:40:08.444423 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:40:08.444423 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:08.444423 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:08.444423 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:40:08.444423 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:08.444423 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:40:08.444423 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:40:08.444423 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:40:08.444423 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:40:08.444423 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:40:08.444423 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:08.444423 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:08.444423 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:08.444423 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:40:08.444423 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:08.444423 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:08.444423 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:08.444423 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:40:08.444797 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:08.444797 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:40:08.444797 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:40:08.444797 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:08.444797 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:40:08.444797 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:08.444797 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:40:08.444797 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:40:08.444797 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:08.444797 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:40:08.444797 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:08.444797 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:40:08.444797 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:40:08.444797 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:40:08.445151 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:40:08.445151 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:40:08.445151 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:40:08.445151 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:08.445905 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:40:08.445905 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:08.445905 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:40:08.445905 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:40:08.445905 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:08.445905 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:40:08.445905 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:08.445905 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:40:08.446451 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:08.446451 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:40:08.446451 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:40:08.446451 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:40:08.446451 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:08.446451 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:40:08.446451 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:08.446451 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:40:08.446451 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:08.446451 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:40:08.446451 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:08.446451 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:40:08.446451 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:08.446451 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:40:08.446451 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:40:08.447033 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:08.447033 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:40:08.447033 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:08.447033 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:40:08.447033 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:40:08.447033 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:08.447033 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:40:08.447033 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:08.447033 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:40:08.447375 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:08.447375 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:40:08.447375 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:08.447375 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:40:08.447375 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:40:08.447375 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:40:08.447375 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:40:08.447375 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:40:08.447375 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:40:08.447375 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:08.447375 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:08.447770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:08.447770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:40:08.447770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:08.447770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:08.447770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:08.447770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:40:08.447770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:08.447770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:40:08.447770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:40:08.447770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:08.447770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:40:08.447770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:08.447770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:40:08.447770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:40:08.448263 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:08.448263 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:40:08.448263 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:08.448263 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:40:08.448458 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:08.448458 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:40:08.448458 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:40:08.448458 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:40:08.448458 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:40:08.448458 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:40:08.448458 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:08.448458 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:40:08.448458 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:40:08.448458 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:08.448458 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:40:08.448458 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:40:08.448458 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:08.448458 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:08.448458 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:40:08.448458 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:08.448458 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:40:08.449069 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:40:08.449069 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:40:08.449069 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:40:08.449069 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:08.449069 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:08.449069 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:08.449069 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:40:08.449069 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:08.449069 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:08.449069 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:08.449069 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:40:08.449069 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:08.449069 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:40:08.449069 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:40:08.449069 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:08.449069 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:40:08.449069 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:08.449069 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:40:08.449069 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:40:08.449732 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:08.449732 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:40:08.449732 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:08.449732 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:40:08.449969 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:40:08.450258 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:40:08.450389 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:40:08.450549 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:40:08.450814 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:40:08.451046 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:40:08.451168 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:08.451650 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:40:08.451650 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:08.451650 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:40:08.451650 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:40:08.451650 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:08.451650 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:40:08.451650 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:08.451650 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:40:08.452571 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:08.452571 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:40:08.452571 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:08.452571 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:40:08.452571 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:40:08.452571 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:40:08.452571 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:40:08.452571 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:08.452571 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:08.452571 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:08.452571 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:40:08.452571 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:08.452851 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:08.452851 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:08.452851 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:40:08.452851 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:08.452851 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:40:08.452851 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:40:08.452851 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:08.452851 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:40:08.452851 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:08.452851 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:40:08.452851 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:40:08.452851 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:08.452851 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:40:08.453079 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:08.453079 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:40:08.453562 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:08.453562 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:40:08.453562 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:40:08.453562 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:40:08.453562 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:40:08.453562 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:40:08.453562 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:08.453562 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:40:08.453562 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:40:08.453562 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:08.453562 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:40:08.453562 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:40:08.453562 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:08.453562 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:08.453562 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:40:08.453562 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:08.453852 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:40:08.453852 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:40:08.453852 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:08.453852 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:08.453852 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:08.453852 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:40:08.453852 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:08.453852 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:08.453852 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:08.453852 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:40:08.453852 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:08.453852 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:40:08.453852 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:40:08.453852 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:08.453852 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:40:08.453852 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:08.453852 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:40:08.453852 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:40:08.453852 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:08.454163 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:40:08.454163 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:08.454163 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:40:12.967570 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:40:12.968467 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:40:12.968467 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:40:12.970004 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:40:13.004473 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:40:13.004937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:13.005946 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:40:13.005946 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:13.005946 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:40:13.005946 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:40:13.005946 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:13.005946 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:40:13.005946 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:13.005946 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:40:13.006996 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.038 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:13.006996 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:40:13.006996 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:13.006996 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:40:13.006996 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:40:13.006996 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:40:13.006996 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:40:13.006996 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:13.006996 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:13.006996 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:13.006996 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:40:13.006996 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:13.007575 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:13.007575 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:13.007575 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:40:13.007575 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:13.007575 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:40:13.007575 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:40:13.007575 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:13.007575 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:40:13.007575 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:13.007575 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:40:13.007575 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:40:13.007575 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:13.007575 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:40:13.008226 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:13.008226 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:40:13.008226 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:13.008226 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:40:13.008226 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:40:13.008226 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:40:13.008226 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:40:13.008226 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:40:13.008226 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:13.008226 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:40:13.008226 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:40:13.008226 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:13.008226 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:40:13.008226 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:40:13.008226 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:13.008931 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:13.008931 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:40:13.008931 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:13.008931 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:40:13.008931 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:40:13.008931 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:13.008931 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:13.008931 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:13.008931 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:40:13.008931 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:13.008931 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:13.008931 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:13.008931 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:40:13.008931 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:13.008931 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:40:13.008931 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:40:13.008931 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:13.008931 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:40:13.008931 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:13.009938 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:40:13.009938 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:40:13.009938 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:13.009938 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:40:13.009938 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:13.009938 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:40:13.432188 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:40:13.432699 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:40:13.432875 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:40:13.433939 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:40:13.434346 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:13.435583 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:40:13.435583 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:13.435583 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:40:13.435583 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:40:13.435583 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:13.435583 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:40:13.435583 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:13.435583 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:40:13.437133 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:13.437133 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:40:13.437133 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:13.437133 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:40:13.437133 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:40:13.437133 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:40:13.437133 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:40:13.437133 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:13.437133 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:13.437133 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:13.437133 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:40:13.437133 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:13.437782 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:13.437782 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:13.437782 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:40:13.437782 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:13.437782 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:40:13.437782 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:40:13.437782 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:13.437782 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:40:13.437782 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:13.437782 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:40:13.437782 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:40:13.437782 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:13.437782 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:40:13.438416 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:13.438416 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:40:13.438837 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:13.438837 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:40:13.438837 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:40:13.438837 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:40:13.438837 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:40:13.438837 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:40:13.438837 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:13.438837 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:40:13.438837 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:40:13.438837 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:13.438837 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:40:13.438837 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:40:13.438837 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:13.438837 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:13.438837 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:40:13.438837 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:13.439643 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:40:13.439643 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:40:13.439643 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:13.439643 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:13.439643 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:13.439643 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:40:13.439643 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:13.439643 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:13.439643 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:13.439643 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:40:13.439643 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:13.439643 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:40:13.439643 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:40:13.439643 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:13.439643 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:40:13.439643 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:13.439643 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:40:13.439643 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:40:13.439643 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:13.440564 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:40:13.440564 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:13.440564 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:40:13.440564 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:40:13.440564 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:40:13.440564 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:40:13.440564 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:40:13.440564 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:40:13.441072 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:13.441517 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:40:13.441517 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:13.441517 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:40:13.441517 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:40:13.441517 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:13.441517 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:40:13.441517 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:13.441517 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:40:13.442313 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:13.442313 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:40:13.442313 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:40:13.442313 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:40:13.442313 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:13.442313 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:40:13.442313 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:13.442313 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:40:13.442313 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:13.442313 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:40:13.442313 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:13.442313 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:40:13.442313 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:13.442313 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:40:13.442313 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:40:13.443105 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:13.443105 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:40:13.443105 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:13.443105 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:40:13.443105 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:40:13.443105 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:13.443105 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:40:13.443105 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:13.443105 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:40:13.443105 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:13.443105 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:40:13.443105 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:13.443105 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:40:13.443105 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:40:13.443866 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:40:13.443866 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:40:13.443866 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:40:13.443866 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:40:13.443866 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:40:13.443866 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:13.443866 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:13.443866 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:13.443866 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:40:13.443866 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:13.443866 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:13.443866 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:13.443866 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:40:13.444514 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:13.444514 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:40:13.444514 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:40:13.444514 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:13.444514 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:40:13.444514 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:13.444514 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:40:13.444514 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:40:13.444514 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:13.444514 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:40:13.444514 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:13.444514 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:40:13.444835 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:13.444835 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:40:13.444835 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:40:13.444835 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:40:13.444835 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:40:13.444835 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:40:13.444835 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:13.444835 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:40:13.444835 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:40:13.444835 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:13.444835 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:40:13.444835 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:40:13.444835 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:13.444835 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:13.444835 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:40:13.444835 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:13.445262 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:40:13.445262 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:40:13.445262 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:40:13.445262 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:40:13.445262 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:40:13.445262 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:13.445262 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:13.445262 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:13.445262 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:40:13.445262 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:13.445262 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:13.445262 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:13.445262 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:40:13.445262 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:13.445262 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:40:13.445262 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:40:13.445262 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:13.445262 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:40:13.445810 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:13.445810 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:40:13.445810 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:40:13.445810 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:13.445810 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:40:13.445810 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:13.445810 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:40:13.445810 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:40:13.445810 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:40:13.445810 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:40:13.446106 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:40:13.446106 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:40:13.446106 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:13.446690 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:40:13.446690 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:13.446690 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:40:13.446690 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:40:13.446690 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:13.446690 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:40:13.446690 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:13.446690 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:40:13.448245 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:13.448245 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:40:13.448245 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:40:13.448245 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:40:13.448245 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:13.448245 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:40:13.448245 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:13.448245 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:40:13.448245 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:13.448245 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:40:13.448245 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:13.448245 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:40:13.448245 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:13.448245 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:40:13.448245 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:40:13.448696 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:13.448696 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:40:13.448696 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:13.448696 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:40:13.448696 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:40:13.448696 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:13.448696 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:40:13.448696 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:13.448696 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:40:13.449833 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:13.449833 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:40:13.449833 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:13.449833 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:40:13.449833 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:40:13.449833 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:40:13.449833 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:40:13.449833 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:40:13.449833 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:40:13.449833 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:13.449833 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:13.450159 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:13.450159 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:40:13.450159 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:13.450159 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:13.450159 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:13.450159 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:40:13.450159 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:13.450159 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:40:13.450159 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:40:13.450159 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:13.450159 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:40:13.450159 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:13.450159 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:40:13.450159 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:40:13.450560 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:13.450560 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:40:13.450560 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:13.450560 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:40:13.451803 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:13.451803 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:40:13.451803 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:40:13.451803 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:40:13.451803 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:40:13.451803 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:40:13.451803 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:13.451803 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:40:13.451803 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:40:13.451803 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:13.451803 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:40:13.451803 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:40:13.451803 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:13.451803 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:13.451803 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:40:13.451803 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:13.451803 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:40:13.452303 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:40:13.452303 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:40:13.452303 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:40:13.452303 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:13.452303 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:13.452303 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:13.452303 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:40:13.452303 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:13.452303 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:13.452303 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:13.452303 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:40:13.452303 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:13.452303 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:40:13.452303 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:40:13.452303 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:13.452303 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:40:13.452303 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:13.452303 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:40:13.452303 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:40:13.452871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:13.452871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:40:13.452871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:13.452871 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:40:13.452871 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:40:13.452871 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:40:13.452871 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:40:13.452871 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:40:13.453293 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:40:13.454163 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:40:13.454420 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:13.455375 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:40:13.455375 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:13.455375 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:40:13.455375 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:40:13.455375 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:13.455375 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:40:13.455375 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:13.455375 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:40:13.456985 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:13.456985 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:40:13.456985 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:13.456985 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:40:13.456985 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:40:13.456985 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:40:13.456985 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:40:13.456985 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:13.456985 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:13.456985 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:13.456985 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:40:13.456985 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:13.457394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:13.457394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:13.457394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:40:13.457394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:13.457394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:40:13.457394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:40:13.457394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:13.457394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:40:13.457394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:13.457394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:40:13.457394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:40:13.457394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:13.457394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:40:13.457743 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:13.457743 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:40:13.458689 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:13.458689 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:40:13.458689 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:40:13.458689 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:40:13.458689 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:40:13.458689 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:40:13.458689 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:13.458689 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:40:13.458689 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:40:13.458689 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:13.458689 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:40:13.458689 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:40:13.458689 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:13.458689 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:13.458689 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:40:13.458689 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:13.459154 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:40:13.459154 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:40:13.459154 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:13.459154 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:13.459154 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:13.459154 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:40:13.459154 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:13.459154 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:13.459154 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:13.459154 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:40:13.459154 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:13.459154 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:40:13.459154 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:40:13.459154 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:13.459154 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:40:13.459154 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:13.459154 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:40:13.459154 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:40:13.459154 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:13.459803 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:40:13.459803 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:13.459803 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:40:17.968234 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:40:17.969063 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:40:17.969171 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:40:17.970420 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:40:18.005943 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:40:18.006362 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:18.007688 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:40:18.007688 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:18.007688 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:40:18.007688 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:40:18.007688 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:18.007688 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:40:18.007688 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:18.007688 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:40:18.008490 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.039 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:18.008490 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:40:18.008490 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:18.008490 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:40:18.008490 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:40:18.008490 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:40:18.008490 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:40:18.008490 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:18.008490 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:18.008490 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:18.008490 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:40:18.008490 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:18.009383 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:18.009383 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:18.009383 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:40:18.009383 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:18.009383 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:40:18.009383 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:40:18.009383 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:18.009383 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:40:18.009383 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:18.009383 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:40:18.009383 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:40:18.009383 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:18.009383 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:40:18.010035 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:18.010035 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:40:18.010035 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:18.010035 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:40:18.010035 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:40:18.010035 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:40:18.010035 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:40:18.010035 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:40:18.010035 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:18.010035 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:40:18.010035 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:40:18.010035 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:18.010035 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:40:18.010035 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:40:18.010035 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:18.010854 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:18.010854 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:40:18.010854 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:18.010854 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:40:18.010854 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:40:18.010854 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:18.010854 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:18.010854 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:18.010854 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:40:18.010854 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:18.010854 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:18.010854 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:18.010854 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:40:18.010854 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:18.010854 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:40:18.010854 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:40:18.010854 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:18.010854 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:40:18.010854 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:18.011868 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:40:18.011868 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:40:18.011868 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:18.011868 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:40:18.011868 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:18.011868 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:40:18.432882 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:40:18.433535 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:40:18.433727 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:40:18.434628 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:40:18.434964 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:18.435840 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:40:18.435840 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:18.435840 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:40:18.435840 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:40:18.435840 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:18.435840 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:40:18.435840 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:18.435840 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:40:18.437808 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:18.437808 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:40:18.437808 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:18.437808 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:40:18.437808 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:40:18.437808 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:40:18.437808 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:40:18.437808 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:18.437808 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:18.437808 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:18.437808 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:40:18.437808 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:18.438044 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:18.438044 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:18.438044 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:40:18.438044 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:18.438044 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:40:18.438044 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:40:18.438044 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:18.438044 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:40:18.438044 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:18.438044 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:40:18.438044 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:40:18.438044 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:18.438044 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:40:18.438503 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:18.438503 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:40:18.439468 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:18.439468 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:40:18.439468 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:40:18.439468 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:40:18.439468 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:40:18.439468 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:40:18.439468 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:18.439468 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:40:18.439468 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:40:18.439468 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:18.439468 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:40:18.439468 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:40:18.439468 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:18.439468 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:18.439468 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:40:18.439468 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:18.439769 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:40:18.439769 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:40:18.439769 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:18.439769 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:18.439769 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:18.439769 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:40:18.439769 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:18.439769 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:18.439769 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:18.439769 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:40:18.439769 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:18.439769 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:40:18.439769 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:40:18.439769 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:18.439769 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:40:18.439769 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:18.439769 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:40:18.439769 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:40:18.439769 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:18.440099 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:40:18.440099 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:18.440099 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:40:18.440099 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:40:18.440183 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:40:18.440376 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:40:18.440622 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:40:18.441089 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:40:18.441451 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:18.442231 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:40:18.442231 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:18.442231 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:40:18.442231 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:40:18.442231 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:18.442231 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:40:18.442231 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:18.442231 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:40:18.442986 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:18.442986 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:40:18.442986 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:40:18.442986 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:40:18.442986 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:18.442986 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:40:18.442986 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:18.442986 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:40:18.442986 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:18.442986 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:40:18.442986 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:18.442986 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:40:18.442986 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:18.442986 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:40:18.442986 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:40:18.443271 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:18.443271 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:40:18.443271 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:18.443271 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:40:18.443271 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:40:18.443271 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:18.443271 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:40:18.443271 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:18.443271 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:40:18.443774 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:18.443774 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:40:18.443774 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:18.443774 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:40:18.443774 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:40:18.443774 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:40:18.443774 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:40:18.443774 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:40:18.443774 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:40:18.443774 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:40:18.443774 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:18.443774 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:18.444011 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:18.444011 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:40:18.444011 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:18.444011 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:18.444011 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:18.444011 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:40:18.444011 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:18.444011 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:40:18.444011 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:40:18.444011 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:18.444011 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:40:18.444011 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:18.444011 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:40:18.444011 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:40:18.444241 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:18.444241 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:40:18.444241 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:18.444241 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:40:18.444818 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:18.444818 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:40:18.444818 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:40:18.444818 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:40:18.444818 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:40:18.444818 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:40:18.444818 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:18.444818 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:40:18.444818 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:40:18.444818 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:18.444818 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:40:18.444818 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:40:18.444818 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:18.444818 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:18.444818 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:40:18.444818 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:18.445095 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:40:18.445095 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:40:18.445095 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:40:18.445095 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:40:18.445095 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:40:18.445095 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:18.445095 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:18.445095 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:18.445095 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:40:18.445095 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:18.445095 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:18.445095 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:18.445095 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:40:18.445095 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:18.445095 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:40:18.445095 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:40:18.445095 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:18.445095 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:40:18.445428 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:18.445428 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:40:18.445428 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:40:18.445428 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:18.445428 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:40:18.445428 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:18.445428 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:40:18.445428 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:40:18.445428 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:40:18.445597 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:40:18.445597 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:40:18.446084 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:40:18.446484 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:18.447170 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:40:18.447170 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:18.447170 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:40:18.447170 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:40:18.447170 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:18.447170 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:40:18.447170 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:18.447170 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:40:18.447923 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:18.447923 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:40:18.447923 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:40:18.447923 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:40:18.447923 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:18.447923 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:40:18.447923 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:18.447923 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:40:18.447923 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:18.447923 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:40:18.447923 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:18.447923 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:40:18.447923 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:18.447923 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:40:18.447923 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:40:18.448181 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:18.448181 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:40:18.448181 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:18.448181 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:40:18.448181 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:40:18.448181 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:18.448181 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:40:18.448181 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:18.448181 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:40:18.448855 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:18.448855 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:40:18.448855 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:18.448855 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:40:18.448855 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:40:18.448855 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:40:18.448855 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:40:18.448855 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:40:18.448855 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:40:18.448855 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:18.448855 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:18.449070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:18.449070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:40:18.449070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:18.449070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:18.449070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:18.449070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:40:18.449070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:18.449070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:40:18.449070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:40:18.449070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:18.449070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:40:18.449070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:18.449070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:40:18.449070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:40:18.449302 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:18.449302 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:40:18.449302 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:18.449302 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:40:18.449986 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:18.449986 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:40:18.449986 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:40:18.449986 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:40:18.449986 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:40:18.449986 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:40:18.449986 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:18.449986 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:40:18.449986 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:40:18.449986 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:18.449986 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:40:18.449986 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:40:18.449986 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:18.449986 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:18.449986 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:40:18.449986 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:18.449986 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:40:18.450290 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:40:18.450290 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:40:18.450290 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:40:18.450290 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:18.450290 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:18.450290 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:18.450290 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:40:18.450290 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:18.450290 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:18.450290 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:18.450290 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:40:18.450290 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:18.450290 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:40:18.450290 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:40:18.450290 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:18.450290 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:40:18.450290 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:18.450290 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:40:18.450290 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:40:18.450629 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:18.450629 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:40:18.450629 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:18.450629 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:40:18.452509 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:40:18.452801 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:40:18.452892 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:40:18.453065 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:40:18.453515 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:40:18.453855 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:40:18.454031 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:18.454727 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:40:18.454727 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:18.454727 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:40:18.454727 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:40:18.454727 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:18.454727 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:40:18.454727 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:18.454727 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:40:18.455568 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:18.455568 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:40:18.455568 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:18.455568 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:40:18.455568 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:40:18.455568 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:40:18.455568 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:40:18.455568 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:18.455568 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:18.455568 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:18.455568 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:40:18.455568 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:18.456633 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:18.456633 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:18.456633 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:40:18.456633 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:18.456633 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:40:18.456633 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:40:18.456633 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:18.456633 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:40:18.456633 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:18.456633 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:40:18.456633 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:40:18.456633 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:18.456633 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:40:18.457135 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:18.457135 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:40:18.457135 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:18.457135 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:40:18.457135 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:40:18.457135 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:40:18.457135 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:40:18.457135 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:40:18.457135 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:18.457135 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:40:18.457135 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:40:18.457135 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:18.457135 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:40:18.457135 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:40:18.457135 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:18.457745 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:18.457745 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:40:18.457745 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:18.457745 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:40:18.457745 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:40:18.457745 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:18.457745 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:18.457745 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:18.457745 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:40:18.457745 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:18.457745 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:18.457745 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:18.457745 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:40:18.457745 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:18.457745 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:40:18.457745 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:40:18.457745 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:18.457745 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:40:18.457745 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:18.458511 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:40:18.458511 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:40:18.458511 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:18.458511 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:40:18.458511 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:18.458511 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:40:22.968936 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:40:22.969931 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:40:22.970140 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:40:22.971577 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:40:23.009229 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:40:23.009756 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:23.010782 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:40:23.010782 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:23.010782 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:40:23.010782 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:40:23.010782 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:23.010782 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:40:23.010782 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:23.010782 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:40:23.011796 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.041 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:23.011796 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:40:23.011796 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:23.011796 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:40:23.011796 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:40:23.011796 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:40:23.011796 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:40:23.011796 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:23.011796 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:23.011796 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:23.011796 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:40:23.011796 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:23.012429 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:23.012429 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:23.012429 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:40:23.012429 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:23.012429 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:40:23.012429 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:40:23.012429 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:23.012429 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:40:23.012429 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:23.012429 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:40:23.012429 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:40:23.012429 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:23.012429 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:40:23.013080 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:23.013080 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:40:23.013080 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:23.013080 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:40:23.013080 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:40:23.013080 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:40:23.013080 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:40:23.013080 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:40:23.013080 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:23.013080 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:40:23.013080 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:40:23.013080 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:23.013080 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:40:23.013080 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:40:23.013080 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:23.013646 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:23.013646 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:40:23.013646 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:23.013646 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:40:23.013646 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:40:23.013646 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:23.013646 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:23.013646 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:23.013646 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:40:23.013646 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:23.013646 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:23.013646 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:23.013646 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:40:23.013646 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:23.013646 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:40:23.013646 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:40:23.013646 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:23.013646 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:40:23.013646 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:23.014288 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:40:23.014288 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:40:23.014288 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:23.014288 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:40:23.014288 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:23.014288 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:40:23.433688 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:40:23.434289 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:40:23.434512 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:40:23.435408 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:40:23.435744 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:23.436627 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:40:23.436627 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:23.436627 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:40:23.436627 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:40:23.436627 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:23.436627 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:40:23.436627 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:23.436627 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:40:23.438174 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:23.438174 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:40:23.438174 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:23.438174 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:40:23.438174 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:40:23.438174 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:40:23.438174 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:40:23.438174 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:23.438174 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:23.438174 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:23.438174 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:40:23.438174 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:23.438481 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:23.438481 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:23.438481 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:40:23.438481 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:23.438481 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:40:23.438481 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:40:23.438481 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:23.438481 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:40:23.438481 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:23.438481 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:40:23.438481 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:40:23.438481 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:23.438481 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:40:23.438740 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:23.438740 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:40:23.439788 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:23.439788 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:40:23.439788 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:40:23.439788 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:40:23.439788 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:40:23.439788 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:40:23.439788 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:23.439788 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:40:23.439788 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:40:23.439788 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:23.439788 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:40:23.439788 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:40:23.439788 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:23.439788 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:23.439788 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:40:23.439788 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:23.440145 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:40:23.440145 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:40:23.440145 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:23.440145 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:23.440145 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:23.440145 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:40:23.440145 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:23.440145 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:23.440145 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:23.440145 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:40:23.440145 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:23.440145 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:40:23.440145 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:40:23.440145 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:23.440145 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:40:23.440145 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:23.440145 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:40:23.440145 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:40:23.440145 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:23.440528 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:40:23.440528 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:23.440528 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:40:23.440528 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:40:23.440739 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:40:23.440866 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:40:23.441075 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:40:23.441741 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:40:23.442060 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:23.442820 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:40:23.442820 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:23.442820 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:40:23.442820 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:40:23.442820 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:23.442820 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:40:23.442820 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:23.442820 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:40:23.443987 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:23.443987 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:40:23.443987 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:40:23.443987 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:40:23.443987 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:23.443987 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:40:23.443987 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:23.443987 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:40:23.443987 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:23.443987 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:40:23.443987 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:23.443987 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:40:23.443987 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:23.443987 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:40:23.443987 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:40:23.444965 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:23.444965 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:40:23.444965 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:23.444965 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:40:23.444965 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:40:23.444965 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:23.444965 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:40:23.444965 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:23.444965 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:40:23.445509 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:23.445509 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:40:23.445509 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:23.445509 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:40:23.445509 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:40:23.445509 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:40:23.445509 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:40:23.445509 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:40:23.445509 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:40:23.445509 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:40:23.445509 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:23.445509 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:23.446119 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:23.446119 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:40:23.446119 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:23.446119 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:23.446119 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:23.446119 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:40:23.446119 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:23.446119 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:40:23.446119 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:40:23.446119 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:23.446119 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:40:23.446119 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:23.446119 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:40:23.446119 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:40:23.446772 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:23.446772 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:40:23.446772 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:23.446772 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:40:23.446772 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:40:23.447044 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:23.447044 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:40:23.447044 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:40:23.447044 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:40:23.447044 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:40:23.447044 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:40:23.447044 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:23.447044 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:40:23.447044 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:40:23.447044 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:23.447044 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:40:23.447044 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:40:23.447044 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:23.447044 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:23.447044 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:40:23.447044 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:23.447762 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:40:23.447762 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:40:23.447762 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:40:23.447762 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:40:23.447762 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:40:23.447762 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:23.447762 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:23.447762 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:23.447762 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:40:23.447762 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:23.447762 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:23.447762 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:23.447762 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:40:23.447762 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:23.447762 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:40:23.447762 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:40:23.447762 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:23.447762 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:40:23.448568 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:23.448568 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:40:23.448568 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:40:23.448568 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:23.448568 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:40:23.448568 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:23.448568 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:40:23.448568 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:40:23.448568 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:40:23.448568 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:40:23.449073 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:40:23.449073 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:23.449073 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:40:23.449073 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:23.449073 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:40:23.449073 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:40:23.449073 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:23.449073 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:40:23.449073 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:23.449073 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:40:23.449073 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:23.449073 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:40:23.449073 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:40:23.449073 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:40:23.449700 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:23.449700 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:40:23.449700 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:23.449700 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:40:23.449700 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:23.449700 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:40:23.449700 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:23.449700 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:40:23.449700 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:23.449700 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:40:23.449700 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:40:23.449700 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:23.449700 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:40:23.449700 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:23.449700 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:40:23.449700 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:40:23.449700 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:23.449700 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:40:23.450577 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:23.450577 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:40:23.450577 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:23.450577 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:40:23.450577 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:23.450577 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:40:23.450577 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:40:23.450577 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:40:23.450577 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:40:23.450577 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:40:23.450577 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:40:23.450577 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:23.451175 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:23.451175 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:23.451175 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:40:23.451175 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:23.451175 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:23.451175 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:23.451175 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:40:23.451175 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:23.451175 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:40:23.451175 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:40:23.451175 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:23.451175 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:40:23.451175 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:23.451175 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:40:23.451885 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:40:23.451885 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:23.451885 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:40:23.451885 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:23.451885 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:40:23.451885 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:23.451885 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:40:23.451885 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:40:23.451885 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:40:23.451885 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:40:23.451885 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:40:23.451885 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:23.451885 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:40:23.451885 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:40:23.451885 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:23.452618 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:40:23.452618 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:40:23.452618 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:23.452618 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:23.452618 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:40:23.452618 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:23.452618 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:40:23.452618 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:40:23.452618 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:40:23.452618 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:40:23.452618 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:23.452618 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:23.452618 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:23.452618 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:40:23.452618 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:23.452618 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:23.452618 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:23.452618 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:40:23.452618 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:23.453659 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:40:23.453659 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:40:23.453659 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:23.453659 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:40:23.453659 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:23.453659 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:40:23.453659 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:40:23.453659 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:23.453659 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:40:23.453659 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:23.453659 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:40:23.453659 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:40:23.453659 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:40:23.454128 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:40:23.454128 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:40:23.454128 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:40:23.454128 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:40:23.454333 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:23.455019 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:40:23.455019 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:23.455019 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:40:23.455019 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:40:23.455019 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:23.455019 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:40:23.455019 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:23.455019 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:40:23.455796 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:23.455796 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:40:23.455796 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:23.455796 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:40:23.455796 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:40:23.455796 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:40:23.455796 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:40:23.455796 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:23.455796 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:23.455796 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:23.455796 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:40:23.455796 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:23.456043 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:23.456043 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:23.456043 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:40:23.456043 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:23.456043 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:40:23.456043 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:40:23.456043 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:23.456043 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:40:23.456043 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:23.456043 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:40:23.456043 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:40:23.456043 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:23.456043 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:40:23.456268 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:23.456268 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:40:23.456735 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:23.456735 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:40:23.456735 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:40:23.456735 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:40:23.456735 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:40:23.456735 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:40:23.456735 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:23.456735 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:40:23.456735 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:40:23.456735 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:23.456735 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:40:23.456735 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:40:23.456735 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:23.456735 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:23.456735 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:40:23.456735 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:23.457055 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:40:23.457055 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:40:23.457055 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:23.457055 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:23.457055 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:23.457055 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:40:23.457055 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:23.457055 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:23.457055 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:23.457055 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:40:23.457055 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:23.457055 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:40:23.457055 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:40:23.457055 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:23.457055 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:40:23.457055 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:23.457055 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:40:23.457055 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:40:23.457055 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:23.457410 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:40:23.457410 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:23.457410 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:40:27.970019 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:40:27.970825 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:40:27.970825 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:40:27.972136 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:40:28.000060 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:40:28.000554 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:28.001447 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:40:28.001447 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:28.001447 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:40:28.001447 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:40:28.001447 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:28.001447 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:40:28.001447 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:28.001447 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:40:28.002200 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.031 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:28.002200 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:40:28.002200 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:28.002200 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:40:28.002200 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:40:28.002200 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:40:28.002200 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:40:28.002200 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:28.002200 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:28.002200 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:28.002200 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:40:28.002200 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:28.002865 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:28.002865 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:28.002865 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:40:28.002865 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:28.002865 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:40:28.002865 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:40:28.002865 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:28.002865 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:40:28.002865 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:28.002865 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:40:28.002865 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:40:28.002865 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:28.002865 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:40:28.003521 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:28.003521 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:40:28.003521 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:28.003521 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:40:28.003521 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:40:28.003521 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:40:28.003521 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:40:28.003521 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:40:28.003521 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:28.003521 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:40:28.003521 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:40:28.003521 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:28.003521 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:40:28.003521 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:40:28.003521 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:28.004204 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:28.004204 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:40:28.004204 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:28.004204 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:40:28.004204 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:40:28.004204 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:28.004204 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:28.004204 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:28.004204 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:40:28.004204 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:28.004204 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:28.004204 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:28.004204 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:40:28.004204 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:28.004204 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:40:28.004204 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:40:28.004204 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:28.004204 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:40:28.004204 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:28.005028 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:40:28.005028 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:40:28.005028 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:28.005028 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:40:28.005028 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:28.005028 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:40:28.434448 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:40:28.435073 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:40:28.435288 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:40:28.436907 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:40:28.437182 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:28.438030 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:40:28.438030 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:28.438030 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:40:28.438030 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:40:28.438030 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:28.438030 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:40:28.438030 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:28.438030 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:40:28.439483 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:28.439483 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:40:28.439483 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:28.439483 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:40:28.439483 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:40:28.439483 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:40:28.439483 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:40:28.439483 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:28.439483 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:28.439483 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:28.439483 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:40:28.439483 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:28.440158 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:28.440158 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:28.440158 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:40:28.440158 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:28.440158 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:40:28.440158 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:40:28.440158 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:28.440158 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:40:28.440158 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:28.440158 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:40:28.440158 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:40:28.440158 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:28.440158 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:40:28.440771 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:28.440771 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:40:28.441127 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:28.441127 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:40:28.441127 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:40:28.441127 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:40:28.441127 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:40:28.441127 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:40:28.441127 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:28.441127 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:40:28.441127 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:40:28.441127 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:28.441127 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:40:28.441127 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:40:28.441127 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:28.441127 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:28.441127 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:40:28.441127 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:28.442056 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:40:28.442056 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:40:28.442056 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:28.442056 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:28.442056 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:28.442056 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:40:28.442056 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:28.442056 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:28.442056 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:28.442056 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:40:28.442056 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:28.442056 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:40:28.442056 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:40:28.442056 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:28.442056 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:40:28.442056 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:28.442056 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:40:28.442056 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:40:28.442056 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:28.442900 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:40:28.442900 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:28.442900 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:40:28.442900 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:40:28.442900 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:40:28.442900 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:40:28.442900 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:40:28.442900 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:40:28.443291 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:28.443948 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:40:28.443948 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:28.443948 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:40:28.443948 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:40:28.443948 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:28.443948 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:40:28.443948 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:28.443948 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:40:28.445227 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:28.445227 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:40:28.445227 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:40:28.445227 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:40:28.445227 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:28.445227 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:40:28.445227 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:28.445227 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:40:28.445227 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:28.445227 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:40:28.445227 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:28.445227 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:40:28.445227 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:28.445227 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:40:28.445227 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:40:28.446881 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:28.446881 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:40:28.446881 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:28.446881 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:40:28.446881 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:40:28.446881 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:28.446881 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:40:28.446881 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:28.446881 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:40:28.446881 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:28.446881 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:40:28.446881 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:28.446881 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:40:28.446881 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:40:28.448552 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:40:28.448552 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:40:28.448552 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:40:28.448552 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:40:28.448552 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:40:28.448552 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:28.448552 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:28.448552 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:28.448552 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:40:28.448552 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:28.448552 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:28.448552 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:28.448552 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:40:28.450035 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:28.450035 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:40:28.450035 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:40:28.450035 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:28.450035 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:40:28.450035 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:28.450035 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:40:28.450035 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:40:28.450035 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:28.450035 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:40:28.450035 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:28.450035 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:40:28.450580 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:28.450580 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:40:28.450580 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:40:28.450580 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:40:28.450580 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:40:28.450580 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:40:28.450580 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:28.450580 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:40:28.450580 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:40:28.450580 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:28.450580 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:40:28.450580 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:40:28.450580 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:28.450580 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:28.450580 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:40:28.450580 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:28.451117 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:40:28.451117 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:40:28.451117 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:40:28.451117 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:40:28.451117 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:40:28.451117 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:28.451117 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:28.451117 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:28.451117 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:40:28.451117 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:28.451117 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:28.451117 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:28.451117 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:40:28.451117 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:28.451117 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:40:28.451117 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:40:28.451117 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:28.451117 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:40:28.451511 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:28.451511 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:40:28.451511 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:40:28.451511 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:28.451511 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:40:28.451511 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:28.451511 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:40:28.451511 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:40:28.451511 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:40:28.451511 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:40:28.451747 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:40:28.451747 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:40:28.451747 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:28.451747 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:40:28.451747 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:28.451747 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:40:28.451747 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:40:28.451747 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:28.451747 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:40:28.451747 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:28.451747 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:40:28.451747 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:28.452000 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:40:28.452000 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:40:28.452000 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:40:28.452000 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:28.452000 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:40:28.452000 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:28.452000 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:40:28.452000 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:28.452000 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:40:28.452000 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:28.452000 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:40:28.452000 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:28.452000 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:40:28.452000 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:40:28.452000 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:28.452000 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:40:28.452000 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:28.452398 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:40:28.452398 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:40:28.452398 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:28.452398 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:40:28.452398 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:28.452398 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:40:28.452398 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:28.452398 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:40:28.452398 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:28.452398 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:40:28.452398 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:40:28.452398 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:40:28.452398 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:40:28.452691 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:40:28.452691 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:40:28.452691 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:28.452691 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:28.452691 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:28.452691 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:40:28.452691 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:28.452691 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:28.452691 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:28.452691 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:40:28.452691 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:28.452691 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:40:28.452691 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:40:28.452958 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:28.452958 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:40:28.452958 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:28.452958 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:40:28.452958 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:40:28.452958 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:28.452958 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:40:28.452958 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:28.452958 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:40:28.452958 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:28.452958 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:40:28.452958 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:40:28.453222 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:40:28.453222 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:40:28.453222 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:40:28.453222 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:28.453222 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:40:28.453222 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:40:28.453222 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:28.453222 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:40:28.453222 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:40:28.453222 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:28.453222 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:28.453222 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:40:28.453222 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:28.453222 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:40:28.453222 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:40:28.453222 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:40:28.453222 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:40:28.453222 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:28.453222 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:28.453700 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:28.453700 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:40:28.453700 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:28.453700 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:28.453700 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:28.453700 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:40:28.453700 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:28.453700 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:40:28.453700 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:40:28.453700 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:28.453700 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:40:28.453700 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:28.453700 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:40:28.453700 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:40:28.453700 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:28.453700 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:40:28.453700 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:28.453700 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:40:28.454108 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:40:28.454108 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:40:28.454108 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:40:28.454108 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:40:28.454295 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:40:28.454516 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:40:28.454601 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:28.455060 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:40:28.455060 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:28.455060 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:40:28.455060 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:40:28.455060 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:28.455060 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:40:28.455060 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:28.455060 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:40:28.455794 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:28.455794 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:40:28.455794 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:28.455794 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:40:28.455794 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:40:28.455794 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:40:28.455794 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:40:28.455794 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:28.455794 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:28.455794 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:28.455794 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:40:28.455794 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:28.456122 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:28.456122 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:28.456122 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:40:28.456122 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:28.456122 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:40:28.456122 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:40:28.456122 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:28.456122 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:40:28.456122 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:28.456122 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:40:28.456122 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:40:28.456122 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:28.456122 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:40:28.456433 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:28.456433 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:40:28.456652 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:28.456652 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:40:28.456652 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:40:28.456652 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:40:28.456652 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:40:28.456652 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:40:28.456652 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:28.456652 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:40:28.456652 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:40:28.456652 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:28.456652 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:40:28.456652 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:40:28.456652 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:28.456652 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:28.456652 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:40:28.456652 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:28.456969 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:40:28.456969 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:40:28.456969 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:28.456969 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:28.456969 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:28.456969 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:40:28.456969 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:28.456969 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:28.456969 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:28.456969 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:40:28.456969 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:28.456969 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:40:28.456969 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:40:28.456969 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:28.456969 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:40:28.456969 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:28.456969 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:40:28.456969 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:40:28.456969 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:28.457359 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:40:28.457359 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:28.457359 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:40:32.970789 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:40:32.971481 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:40:32.971617 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:40:32.972938 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:40:33.010293 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:40:33.010629 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:33.011580 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:40:33.011580 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:33.011580 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:40:33.011580 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:40:33.011580 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:33.011580 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:40:33.011580 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:33.011580 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:40:33.012415 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.040 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:33.012415 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:40:33.012415 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:33.012415 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:40:33.012415 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:40:33.012415 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:40:33.012415 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:40:33.012415 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:33.012415 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:33.012415 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:33.012415 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:40:33.012415 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:33.013091 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:33.013091 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:33.013091 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:40:33.013091 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:33.013091 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:40:33.013091 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:40:33.013091 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:33.013091 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:40:33.013091 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:33.013091 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:40:33.013091 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:40:33.013091 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:33.013091 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:40:33.014299 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:33.014299 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:40:33.014299 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:33.014299 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:40:33.014299 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:40:33.014299 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:40:33.014299 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:40:33.014299 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:40:33.014299 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:33.014299 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:40:33.014299 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:40:33.014299 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:33.014299 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:40:33.014299 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:40:33.014299 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:33.018113 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:33.018113 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:40:33.018113 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:33.018113 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:40:33.018113 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:40:33.018113 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:33.018113 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:33.018113 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:33.018113 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:40:33.018113 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:33.018113 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:33.018113 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:33.018113 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:40:33.018113 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:33.018113 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:40:33.018113 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:40:33.018113 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:33.018113 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:40:33.018113 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:33.018925 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:40:33.018925 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:40:33.018925 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:33.018925 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:40:33.018925 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:33.018925 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:40:33.435148 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:40:33.435834 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:40:33.436045 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:40:33.437488 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:40:33.437866 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:33.438899 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:40:33.438899 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:33.438899 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:40:33.438899 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:40:33.438899 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:33.438899 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:40:33.438899 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:33.438899 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:40:33.440940 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:33.440940 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:40:33.440940 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:33.440940 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:40:33.440940 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:40:33.440940 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:40:33.440940 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:40:33.440940 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:33.440940 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:33.440940 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:33.440940 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:40:33.440940 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:33.441528 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:33.441528 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:33.441528 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:40:33.441528 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:33.441528 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:40:33.441528 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:40:33.441528 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:33.441528 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:40:33.441528 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:33.441528 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:40:33.441528 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:40:33.441528 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:33.441528 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:40:33.442053 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:33.442053 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:40:33.442794 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:33.442794 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:40:33.442794 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:40:33.442794 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:40:33.442794 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:40:33.442794 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:40:33.442794 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:33.442794 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:40:33.442794 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:40:33.442794 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:33.442794 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:40:33.442794 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:40:33.442794 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:33.442794 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:33.442794 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:40:33.442794 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:33.443521 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:40:33.443521 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:40:33.443521 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:33.443521 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:33.443521 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:33.443521 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:40:33.443521 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:33.443521 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:33.443521 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:33.443521 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:40:33.443521 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:33.443521 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:40:33.443521 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:40:33.443521 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:33.443521 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:40:33.443521 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:33.443521 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:40:33.443521 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:40:33.443521 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:33.444272 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:40:33.444272 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:33.444272 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:40:33.444272 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:40:33.444272 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:40:33.444272 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:40:33.444272 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:40:33.444272 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:40:33.444811 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:33.445154 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:40:33.445154 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:33.445154 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:40:33.445154 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:40:33.445154 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:33.445154 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:40:33.445154 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:33.445154 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:40:33.445925 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:33.445925 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:40:33.445925 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:40:33.445925 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:40:33.445925 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:33.445925 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:40:33.445925 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:33.445925 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:40:33.445925 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:33.445925 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:40:33.445925 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:33.445925 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:40:33.445925 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:33.445925 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:40:33.445925 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:40:33.446341 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:33.446341 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:40:33.446341 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:33.446341 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:40:33.446341 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:40:33.446341 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:33.446341 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:40:33.446341 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:33.446341 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:40:33.446766 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:33.446766 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:40:33.446766 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:33.446766 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:40:33.446766 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:40:33.446766 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:40:33.446766 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:40:33.446766 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:40:33.446766 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:40:33.446766 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:40:33.446766 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:33.446766 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:33.447097 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:33.447097 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:40:33.447097 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:33.447097 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:33.447097 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:33.447097 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:40:33.447097 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:33.447097 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:40:33.447097 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:40:33.447097 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:33.447097 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:40:33.447097 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:33.447097 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:40:33.447097 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:40:33.447472 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:33.447472 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:40:33.447472 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:33.447472 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:40:33.447799 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:33.447799 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:40:33.447799 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:40:33.447799 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:40:33.447799 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:40:33.447799 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:40:33.447799 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:33.447799 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:40:33.447799 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:40:33.447799 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:33.447799 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:40:33.447799 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:40:33.447799 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:33.447799 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:33.447799 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:40:33.447799 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:33.448205 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:40:33.448205 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:40:33.448205 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:40:33.448205 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:40:33.448205 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:40:33.448205 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:33.448205 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:33.448205 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:33.448205 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:40:33.448205 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:33.448205 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:33.448205 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:33.448205 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:40:33.448205 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:33.448205 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:40:33.448205 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:40:33.448205 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:33.448205 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:40:33.448750 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:33.448750 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:40:33.448750 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:40:33.448750 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:33.448750 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:40:33.448750 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:33.448750 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:40:33.448750 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:40:33.448750 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:40:33.448750 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:40:33.449035 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:40:33.449244 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:40:33.449621 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:33.450393 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:40:33.450393 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:33.450393 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:40:33.450393 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:40:33.450393 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:33.450393 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:40:33.450393 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:33.450393 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:40:33.451155 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:33.451155 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:40:33.451155 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:40:33.451155 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:40:33.451155 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:33.451155 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:40:33.451155 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:33.451155 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:40:33.451155 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:33.451155 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:40:33.451155 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:33.451155 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:40:33.451155 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:33.451155 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:40:33.451155 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:40:33.451579 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:33.451579 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:40:33.451579 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:33.451579 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:40:33.451579 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:40:33.451579 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:33.451579 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:40:33.451579 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:33.451579 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:40:33.452036 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:33.452036 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:40:33.452036 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:33.452036 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:40:33.452036 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:40:33.452036 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:40:33.452036 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:40:33.452036 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:40:33.452036 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:40:33.452036 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:33.452036 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:33.452388 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:33.452388 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:40:33.452388 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:33.452388 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:33.452388 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:33.452388 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:40:33.452388 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:33.452388 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:40:33.452388 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:40:33.452388 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:33.452388 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:40:33.452388 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:33.452388 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:40:33.452388 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:40:33.452750 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:33.452750 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:40:33.452750 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:33.452750 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:40:33.453166 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:33.453166 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:40:33.453166 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:40:33.453166 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:40:33.453166 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:40:33.453166 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:40:33.453166 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:33.453166 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:40:33.453166 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:40:33.453166 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:33.453166 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:40:33.453166 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:40:33.453166 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:33.453166 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:33.453166 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:40:33.453166 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:33.453166 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:40:33.453666 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:40:33.453666 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:40:33.453666 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:40:33.453666 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:33.453666 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:33.453666 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:33.453666 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:40:33.453666 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:33.453666 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:33.453666 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:33.453666 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:40:33.453666 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:33.453666 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:40:33.453666 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:40:33.453666 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:33.453666 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:40:33.453666 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:33.453666 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:40:33.453666 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:40:33.454235 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:33.454235 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:40:33.454235 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:33.454235 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:40:33.454235 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:40:33.454235 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:40:33.454441 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:40:33.454479 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:40:33.454928 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:40:33.455259 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:40:33.455523 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:33.456195 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:40:33.456195 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:33.456195 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:40:33.456195 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:40:33.456195 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:33.456195 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:40:33.456195 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:33.456195 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:40:33.457125 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:33.457125 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:40:33.457125 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:33.457125 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:40:33.457125 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:40:33.457125 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:40:33.457125 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:40:33.457125 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:33.457125 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:33.457125 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:33.457125 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:40:33.457125 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:33.457473 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:33.457473 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:33.457473 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:40:33.457473 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:33.457473 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:40:33.457473 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:40:33.457473 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:33.457473 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:40:33.457473 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:33.457473 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:40:33.457473 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:40:33.457473 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:33.457473 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:40:33.457798 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:33.457798 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:40:33.458177 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:33.458177 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:40:33.458177 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:40:33.458177 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:40:33.458177 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:40:33.458177 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:40:33.458177 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:33.458177 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:40:33.458177 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:40:33.458177 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:33.458177 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:40:33.458177 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:40:33.458177 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:33.458177 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:33.458177 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:40:33.458177 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:33.458602 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:40:33.458602 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:40:33.458602 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:33.458602 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:33.458602 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:33.458602 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:40:33.458602 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:33.458602 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:33.458602 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:33.458602 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:40:33.458602 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:33.458602 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:40:33.458602 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:40:33.458602 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:33.458602 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:40:33.458602 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:33.458602 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:40:33.458602 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:40:33.458602 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:33.459082 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:40:33.459082 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:33.459082 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:40:37.971479 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:40:37.972427 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:40:37.972427 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:40:37.973740 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:40:38.008299 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:40:38.008758 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:38.009706 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:40:38.009706 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:38.009706 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:40:38.009706 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:40:38.009706 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:38.009706 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:40:38.009706 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:38.009706 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:40:38.010868 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.037 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:38.010868 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:40:38.010868 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:38.010868 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:40:38.010868 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:40:38.010868 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:40:38.010868 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:40:38.010868 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:38.010868 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:38.010868 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:38.010868 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:40:38.010868 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:38.011510 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:38.011510 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:38.011510 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:40:38.011510 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:38.011510 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:40:38.011510 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:40:38.011510 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:38.011510 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:40:38.011510 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:38.011510 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:40:38.011510 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:40:38.011510 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:38.011510 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:40:38.012049 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:38.012049 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:40:38.012049 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:38.012049 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:40:38.012049 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:40:38.012049 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:40:38.012049 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:40:38.012049 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:40:38.012049 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:38.012049 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:40:38.012049 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:40:38.012049 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:38.012049 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:40:38.012049 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:40:38.012049 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:38.013399 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:38.013399 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:40:38.013399 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:38.013399 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:40:38.013399 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:40:38.013399 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:38.013399 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:38.013399 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:38.013399 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:40:38.013399 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:38.013399 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:38.013399 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:38.013399 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:40:38.013399 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:38.013399 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:40:38.013399 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:40:38.013399 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:38.013399 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:40:38.013399 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:38.014164 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:40:38.014164 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:40:38.014164 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:38.014164 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:40:38.014164 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:38.014164 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:40:38.435896 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:40:38.436539 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:40:38.436686 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:40:38.437751 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:40:38.438075 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:38.439005 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:40:38.439005 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:38.439005 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:40:38.439005 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:40:38.439005 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:38.439005 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:40:38.439005 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:38.439005 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:40:38.440114 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:38.440114 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:40:38.440114 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:38.440114 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:40:38.440114 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:40:38.440114 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:40:38.440114 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:40:38.440114 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:38.440114 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:38.440114 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:38.440114 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:40:38.440114 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:38.440776 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:38.440776 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:38.440776 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:40:38.440776 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:38.440776 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:40:38.440776 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:40:38.440776 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:38.440776 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:40:38.440776 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:38.440776 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:40:38.440776 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:40:38.440776 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:38.440776 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:40:38.441096 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:38.441096 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:40:38.441096 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:38.441096 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:40:38.441096 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:40:38.441096 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:40:38.441096 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:40:38.441096 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:40:38.441096 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:38.441096 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:40:38.441096 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:40:38.441096 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:38.441096 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:40:38.441096 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:40:38.441096 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:38.441496 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:38.441496 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:40:38.441496 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:38.441496 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:40:38.441496 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:40:38.441496 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:38.441496 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:38.441496 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:38.441496 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:40:38.441496 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:38.441496 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:38.441496 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:38.441496 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:40:38.441496 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:38.441496 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:40:38.441496 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:40:38.441496 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:38.441496 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:40:38.441496 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:38.441995 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:40:38.441995 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:40:38.441995 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:38.441995 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:40:38.441995 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:38.441995 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:40:38.443543 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:40:38.443828 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:40:38.443960 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:40:38.444078 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:40:38.444755 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:40:38.445075 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:38.445856 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:40:38.445856 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:38.445856 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:40:38.445856 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:40:38.445856 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:38.445856 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:40:38.445856 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:38.445856 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:40:38.446716 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:38.446716 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:40:38.446716 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:40:38.446716 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:40:38.446716 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:38.446716 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:40:38.446716 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:38.446716 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:40:38.446716 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:38.446716 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:40:38.446716 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:38.446716 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:40:38.446716 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:38.446716 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:40:38.446716 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:40:38.447504 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:38.447504 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:40:38.447504 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:38.447504 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:40:38.447504 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:40:38.447504 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:38.447504 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:40:38.447504 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:38.447504 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:40:38.447886 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:38.447886 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:40:38.447886 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:38.447886 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:40:38.447886 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:40:38.447886 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:40:38.447886 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:40:38.447886 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:40:38.447886 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:40:38.447886 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:40:38.447886 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:38.447886 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:38.448408 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:38.448408 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:40:38.448408 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:38.448408 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:38.448408 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:38.448408 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:40:38.448408 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:38.448408 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:40:38.448408 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:40:38.448408 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:38.448408 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:40:38.448408 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:38.448408 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:40:38.448408 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:40:38.448951 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:38.448951 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:40:38.448951 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:38.448951 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:40:38.448951 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:38.448951 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:40:38.448951 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:40:38.448951 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:40:38.448951 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:40:38.448951 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:40:38.448951 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:38.448951 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:40:38.448951 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:40:38.448951 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:38.449568 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:40:38.449568 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:40:38.449568 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:38.449568 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:38.449568 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:40:38.449568 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:38.449568 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:40:38.449568 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:40:38.449568 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:40:38.449568 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:40:38.449568 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:40:38.449568 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:38.449568 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:38.449568 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:38.449568 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:40:38.449568 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:38.449568 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:38.449568 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:38.449568 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:40:38.450317 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:38.450317 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:40:38.450317 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:40:38.450317 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:38.450317 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:40:38.450317 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:38.450317 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:40:38.450317 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:40:38.450317 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:38.450317 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:40:38.450317 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:38.450317 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:40:38.450317 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:40:38.450317 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:40:38.451950 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:40:38.451950 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:40:38.451950 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:40:38.451950 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:38.451950 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:40:38.451950 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:38.451950 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:40:38.451950 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:40:38.451950 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:38.451950 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:40:38.451950 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:38.451950 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:40:38.452281 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:38.452281 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:40:38.452281 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:40:38.452281 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:40:38.452281 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:38.452281 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:40:38.452281 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:38.452281 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:40:38.452281 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:38.452281 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:40:38.452281 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:38.452281 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:40:38.452281 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:38.452281 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:40:38.452281 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:40:38.453011 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:38.453011 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:40:38.453011 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:38.453011 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:40:38.453011 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:40:38.453011 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:38.453011 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:40:38.453011 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:38.453011 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:40:38.453514 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:38.453514 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:40:38.453514 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:38.453514 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:40:38.453514 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:40:38.453514 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:40:38.453514 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:40:38.453514 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:40:38.453514 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:40:38.453514 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:38.453514 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:38.454112 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:38.454112 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:40:38.454112 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:38.454112 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:38.454112 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:38.454112 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:40:38.454112 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:38.454112 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:40:38.454112 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:40:38.454112 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:38.454112 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:40:38.454112 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:38.454112 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:40:38.454112 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:40:38.454902 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:38.454902 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:40:38.454902 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:38.454902 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:40:38.454902 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:38.454902 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:40:38.454902 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:40:38.454902 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:40:38.454902 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:40:38.454902 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:40:38.454902 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:38.454902 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:40:38.454902 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:40:38.454902 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:38.455845 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:40:38.455845 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:40:38.455845 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:38.455845 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:38.455845 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:40:38.455845 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:38.455845 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:40:38.455845 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:40:38.455845 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:40:38.455845 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:40:38.455845 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:38.455845 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:38.455845 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:38.455845 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:40:38.455845 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:38.455845 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:38.455845 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:38.455845 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:40:38.455845 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:38.457065 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:40:38.457065 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:40:38.457065 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:38.457065 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:40:38.457065 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:38.457065 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:40:38.457065 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:40:38.457065 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:38.457065 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:40:38.457065 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:38.457065 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:40:38.457065 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:40:38.457065 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:40:38.457807 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:40:38.457807 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:40:38.457807 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:40:38.457807 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:40:38.457807 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:38.457807 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:40:38.457807 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:38.457807 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:40:38.457807 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:40:38.458128 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:38.458128 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:40:38.458128 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:38.458128 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:40:38.458128 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:38.458128 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:40:38.458128 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:38.458128 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:40:38.458128 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:40:38.458128 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:40:38.458128 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:40:38.458128 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:38.458128 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:38.458128 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:38.458544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:40:38.458544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:38.458544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:38.458544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:38.458544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:40:38.458544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:38.458544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:40:38.458544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:40:38.458544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:38.458544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:40:38.458544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:38.458544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:40:38.458544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:40:38.458544 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:38.459353 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:40:38.459353 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:38.459353 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:40:38.459353 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:38.459353 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:40:38.459353 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:40:38.459353 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:40:38.459353 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:40:38.459353 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:40:38.459353 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:38.459353 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:40:38.459353 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:40:38.459353 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:38.459353 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:40:38.459353 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:40:38.459353 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:38.460115 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:38.460115 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:40:38.460115 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:38.460115 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:40:38.460115 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:40:38.460115 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:38.460115 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:38.460115 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:38.460115 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:40:38.460115 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:38.460115 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:38.460115 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:38.460115 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:40:38.460115 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:38.460115 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:40:38.460115 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:40:38.460115 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:38.460115 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:40:38.460115 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:38.460564 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:40:38.460564 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:40:38.460564 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:38.460564 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:40:38.460564 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:38.460564 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:40:42.972143 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:40:42.973441 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:40:42.973441 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:40:42.974140 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:40:43.007728 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:40:43.008179 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:43.009253 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:40:43.009253 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:43.009253 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:40:43.009253 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:40:43.009253 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:43.009253 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:40:43.009253 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:43.009253 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:40:43.010065 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.036 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:43.010065 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:40:43.010065 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:43.010065 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:40:43.010065 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:40:43.010065 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:40:43.010065 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:40:43.010065 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:43.010065 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:43.010065 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:43.010065 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:40:43.010065 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:43.010474 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:43.010474 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:43.010474 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:40:43.010474 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:43.010474 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:40:43.010474 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:40:43.010474 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:43.010474 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:40:43.010474 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:43.010474 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:40:43.010474 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:40:43.010474 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:43.010474 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:40:43.010828 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:43.010828 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:40:43.010828 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:43.010828 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:40:43.010828 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:40:43.010828 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:40:43.010828 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:40:43.010828 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:40:43.010828 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:43.010828 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:40:43.010828 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:40:43.010828 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:43.010828 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:40:43.010828 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:40:43.010828 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:43.011268 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:43.011268 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:40:43.011268 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:43.011268 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:40:43.011268 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:40:43.011268 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:43.011268 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:43.011268 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:43.011268 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:40:43.011268 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:43.011268 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:43.011268 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:43.011268 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:40:43.011268 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:43.011268 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:40:43.011268 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:40:43.011268 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:43.011268 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:40:43.011268 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:43.011930 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:40:43.011930 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:40:43.011930 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:43.011930 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:40:43.011930 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:43.011930 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:40:43.436670 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:40:43.437348 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:40:43.437591 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:40:43.439008 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:40:43.439488 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:43.440575 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:40:43.440575 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:43.440575 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:40:43.440575 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:40:43.440575 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:43.440575 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:40:43.440575 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:43.440575 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:40:43.442554 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:43.442554 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:40:43.442554 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:43.442554 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:40:43.442554 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:40:43.442554 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:40:43.442554 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:40:43.442554 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:43.442554 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:43.442554 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:43.442554 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:40:43.442554 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:43.442914 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:43.442914 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:43.442914 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:40:43.442914 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:43.442914 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:40:43.442914 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:40:43.442914 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:43.442914 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:40:43.442914 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:43.442914 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:40:43.442914 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:40:43.442914 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:43.442914 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:40:43.443229 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:43.443229 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:40:43.444298 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:43.444298 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:40:43.444298 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:40:43.444298 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:40:43.444298 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:40:43.444298 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:40:43.444298 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:43.444298 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:40:43.444298 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:40:43.444298 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:43.444298 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:40:43.444298 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:40:43.444298 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:43.444298 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:43.444298 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:40:43.444298 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:43.444969 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:40:43.444969 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:40:43.444969 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:43.444969 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:43.444969 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:43.444969 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:40:43.444969 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:43.444969 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:43.444969 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:43.444969 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:40:43.444969 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:43.444969 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:40:43.444969 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:40:43.444969 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:43.444969 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:40:43.444969 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:43.444969 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:40:43.444969 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:40:43.444969 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:43.445487 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:40:43.445487 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:43.445487 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:40:43.445487 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:40:43.445487 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:40:43.445487 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:40:43.445487 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:40:43.445877 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:40:43.446196 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:43.447018 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:40:43.447018 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:43.447018 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:40:43.447018 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:40:43.447018 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:43.447018 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:40:43.447018 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:43.447018 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:40:43.447903 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:43.447903 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:40:43.447903 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:40:43.447903 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:40:43.447903 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:43.447903 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:40:43.447903 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:43.447903 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:40:43.447903 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:43.447903 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:40:43.447903 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:43.447903 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:40:43.447903 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:43.447903 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:40:43.447903 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:40:43.449263 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:43.449263 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:40:43.449263 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:43.449263 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:40:43.449263 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:40:43.449263 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:43.449263 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:40:43.449263 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:43.449263 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:40:43.449263 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:43.449263 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:40:43.449263 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:43.449263 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:40:43.449263 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:40:43.449950 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:40:43.449950 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:40:43.449950 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:40:43.449950 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:40:43.449950 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:40:43.449950 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:43.449950 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:43.449950 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:43.449950 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:40:43.449950 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:43.449950 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:43.449950 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:43.449950 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:40:43.450486 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:43.450486 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:40:43.450486 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:40:43.450486 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:43.450486 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:40:43.450486 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:43.450486 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:40:43.450486 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:40:43.450486 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:43.450486 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:40:43.450486 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:43.450486 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:40:43.450980 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:43.450980 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:40:43.450980 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:40:43.450980 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:40:43.450980 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:40:43.450980 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:40:43.450980 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:43.450980 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:40:43.450980 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:40:43.450980 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:43.450980 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:40:43.450980 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:40:43.450980 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:43.450980 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:43.450980 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:40:43.450980 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:43.451741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:40:43.451741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:40:43.451741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:40:43.451741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:40:43.451741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:40:43.451741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:43.451741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:43.451741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:43.451741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:40:43.451741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:43.451741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:43.451741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:43.451741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:40:43.451741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:43.451741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:40:43.451741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:40:43.451741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:43.451741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:40:43.452670 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:43.452670 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:40:43.452670 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:40:43.452670 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:43.452670 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:40:43.452670 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:43.452670 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:40:43.452670 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:40:43.452670 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:40:43.452670 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:40:43.453067 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:40:43.453067 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:40:43.453067 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:43.453067 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:40:43.453067 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:43.453067 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:40:43.453067 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:40:43.453067 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:43.453067 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:40:43.453067 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:43.453067 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:40:43.453772 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:43.453772 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:40:43.453772 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:40:43.453772 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:40:43.453772 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:43.453772 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:40:43.453772 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:43.453772 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:40:43.453772 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:43.453772 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:40:43.453772 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:43.453772 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:40:43.453772 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:43.453772 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:40:43.453772 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:40:43.454179 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:43.454179 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:40:43.454179 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:43.454179 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:40:43.454179 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:40:43.454179 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:43.454179 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:40:43.454179 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:43.454179 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:40:43.454687 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:43.454687 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:40:43.454687 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:43.454687 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:40:43.454687 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:40:43.454687 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:40:43.454687 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:40:43.454687 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:40:43.454687 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:40:43.454687 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:43.454687 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:43.455030 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:43.455030 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:40:43.455030 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:43.455030 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:43.455030 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:43.455030 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:40:43.455030 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:43.455030 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:40:43.455030 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:40:43.455030 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:43.455030 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:40:43.455030 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:43.455030 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:40:43.455030 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:40:43.455415 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:43.455415 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:40:43.455415 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:43.455415 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:40:43.455918 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:43.455918 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:40:43.455918 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:40:43.455918 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:40:43.455918 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:40:43.455918 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:40:43.455918 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:43.455918 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:40:43.455918 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:40:43.455918 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:43.455918 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:40:43.455918 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:40:43.455918 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:43.455918 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:43.455918 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:40:43.455918 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:43.455918 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:40:43.456477 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:40:43.456477 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:40:43.456477 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:40:43.456477 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:43.456477 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:43.456477 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:43.456477 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:40:43.456477 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:43.456477 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:43.456477 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:43.456477 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:40:43.456477 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:43.456477 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:40:43.456477 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:40:43.456477 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:43.456477 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:40:43.456477 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:43.456477 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:40:43.456477 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:40:43.457377 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:43.457377 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:40:43.457377 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:43.457377 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:40:43.457377 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:40:43.457377 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:40:43.457377 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:40:43.457377 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:40:43.457634 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:40:43.457634 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:40:43.457804 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:43.458603 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:40:43.458603 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:43.458603 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:40:43.458603 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:40:43.458603 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:43.458603 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:40:43.458603 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:43.458603 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:40:43.459519 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:43.459519 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:40:43.459519 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:43.459519 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:40:43.459519 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:40:43.459519 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:40:43.459519 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:40:43.459519 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:43.459519 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:43.459519 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:43.459519 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:40:43.459519 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:43.459778 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:43.459778 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:43.459778 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:40:43.459778 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:43.459778 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:40:43.459778 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:40:43.459778 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:43.459778 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:40:43.459778 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:43.459778 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:40:43.459778 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:40:43.459778 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:43.459778 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:40:43.460022 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:43.460022 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:40:43.460536 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:43.460536 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:40:43.460536 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:40:43.460536 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:40:43.460536 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:40:43.460536 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:40:43.460536 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:43.460536 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:40:43.460536 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:40:43.460536 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:43.460536 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:40:43.460536 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:40:43.460536 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:43.460536 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:43.460536 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:40:43.460536 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:43.460882 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:40:43.460882 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:40:43.460882 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:43.460882 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:43.460882 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:43.460882 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:40:43.460882 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:43.460882 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:43.460882 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:43.460882 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:40:43.460882 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:43.460882 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:40:43.460882 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:40:43.460882 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:43.460882 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:40:43.460882 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:43.460882 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:40:43.460882 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:40:43.460882 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:43.461252 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:40:43.461252 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:43.461252 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:40:47.972859 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:40:47.973773 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:40:47.973773 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:40:47.975088 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:40:47.981714 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG dbcounter [-] [106123] Writing DB stats neutron:SELECT=241 {{(pid=106123) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} May 18 18:40:48.009668 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:40:48.010065 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:48.010789 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:40:48.010789 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:48.010789 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:40:48.010789 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:40:48.010789 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:48.010789 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:40:48.010789 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:48.010789 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:40:48.011461 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.037 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:48.011461 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:40:48.011461 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:48.011461 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:40:48.011461 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:40:48.011461 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:40:48.011461 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:40:48.011461 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:48.011461 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:48.011461 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:48.011461 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:40:48.011461 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:48.012660 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:48.012660 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:48.012660 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:40:48.012660 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:48.012660 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:40:48.012660 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:40:48.012660 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:48.012660 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:40:48.012660 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:48.012660 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:40:48.012660 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:40:48.012660 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:48.012660 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:40:48.013402 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:48.013402 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:40:48.013402 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:48.013402 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:40:48.013402 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:40:48.013402 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:40:48.013402 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:40:48.013402 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:40:48.013402 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:48.013402 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:40:48.013402 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:40:48.013402 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:48.013402 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:40:48.013402 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:40:48.013402 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:48.014190 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:48.014190 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:40:48.014190 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:48.014190 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:40:48.014190 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:40:48.014190 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:48.014190 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:48.014190 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:48.014190 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:40:48.014190 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:48.014190 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:48.014190 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:48.014190 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:40:48.014190 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:48.014190 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:40:48.014190 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:40:48.014190 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:48.014190 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:40:48.014190 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:48.015047 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:40:48.015047 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:40:48.015047 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:48.015047 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:40:48.015047 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:48.015047 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:40:48.437659 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:40:48.438204 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:40:48.438417 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:40:48.439485 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:40:48.439827 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:48.440735 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:40:48.440735 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:48.440735 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:40:48.440735 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:40:48.440735 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:48.440735 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:40:48.440735 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:48.440735 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:40:48.442251 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:48.442251 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:40:48.442251 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:48.442251 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:40:48.442251 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:40:48.442251 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:40:48.442251 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:40:48.442251 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:48.442251 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:48.442251 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:48.442251 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:40:48.442251 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:48.442851 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:48.442851 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:48.442851 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:40:48.442851 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:48.442851 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:40:48.442851 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:40:48.442851 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:48.442851 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:40:48.442851 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:48.442851 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:40:48.442851 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:40:48.442851 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:48.442851 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:40:48.443399 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:48.443399 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:40:48.443927 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:48.443927 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:40:48.443927 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:40:48.443927 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:40:48.443927 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:40:48.443927 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:40:48.443927 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:48.443927 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:40:48.443927 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:40:48.443927 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:48.443927 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:40:48.443927 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:40:48.443927 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:48.443927 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:48.443927 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:40:48.443927 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:48.445747 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:40:48.445747 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:40:48.445747 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:48.445747 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:48.445747 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:48.445747 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:40:48.445747 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:48.445747 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:48.445747 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:48.445747 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:40:48.445747 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:48.445747 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:40:48.445747 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:40:48.445747 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:48.445747 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:40:48.445747 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:48.445747 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:40:48.445747 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:40:48.445747 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:48.447291 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:40:48.447291 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:48.447291 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:40:48.447291 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:40:48.447291 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:40:48.447291 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:40:48.447291 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:40:48.447291 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:40:48.447725 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:48.447725 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:40:48.447725 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:48.447725 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:40:48.447725 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:40:48.447725 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:48.447725 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:40:48.447725 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:48.447725 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:40:48.448491 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:48.448491 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:40:48.448491 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:40:48.448491 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:40:48.448491 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:48.448491 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:40:48.448491 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:48.448491 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:40:48.448491 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:48.448491 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:40:48.448491 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:48.448491 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:40:48.448491 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:48.448491 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:40:48.448491 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:40:48.449863 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:48.449863 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:40:48.449863 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:48.449863 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:40:48.449863 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:40:48.449863 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:48.449863 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:40:48.449863 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:48.449863 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:40:48.449863 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:48.449863 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:40:48.449863 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:48.449863 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:40:48.449863 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:40:48.450861 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:40:48.450861 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:40:48.450861 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:40:48.450861 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:40:48.450861 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:40:48.450861 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:48.450861 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:48.450861 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:48.450861 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:40:48.450861 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:48.450861 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:48.450861 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:48.450861 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:40:48.451293 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:48.451293 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:40:48.451293 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:40:48.451293 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:48.451293 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:40:48.451293 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:48.451293 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:40:48.451293 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:40:48.451293 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:48.451293 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:40:48.451293 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:48.451293 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:40:48.451522 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:48.451522 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:40:48.451522 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:40:48.451522 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:40:48.451522 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:40:48.451522 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:40:48.451522 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:48.451522 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:40:48.451522 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:40:48.451522 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:48.451522 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:40:48.451522 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:40:48.451522 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:48.451522 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:48.451522 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:40:48.451522 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:48.451826 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:40:48.451826 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:40:48.451826 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:40:48.451826 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:40:48.451826 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:40:48.451826 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:48.451826 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:48.451826 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:48.451826 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:40:48.451826 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:48.451826 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:48.451826 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:48.451826 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:40:48.451826 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:48.451826 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:40:48.451826 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:40:48.451826 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:48.451826 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:40:48.452165 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:48.452165 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:40:48.452165 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:40:48.452165 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:48.452165 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:40:48.452165 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:48.452165 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:40:48.452165 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:40:48.452165 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:40:48.452165 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:40:48.452393 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:40:48.452393 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:40:48.452393 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:48.453036 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:40:48.453036 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:48.453036 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:40:48.453036 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:40:48.453036 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:48.453036 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:40:48.453036 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:48.453036 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:40:48.453786 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:48.453786 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:40:48.453786 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:40:48.453786 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:40:48.453786 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:48.453786 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:40:48.453786 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:48.453786 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:40:48.453786 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:48.453786 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:40:48.453786 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:48.453786 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:40:48.453786 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:48.453786 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:40:48.453786 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:40:48.454092 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:48.454092 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:40:48.454092 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:48.454092 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:40:48.454092 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:40:48.454092 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:48.454092 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:40:48.454092 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:48.454092 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:40:48.454696 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:48.454696 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:40:48.454696 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:48.454696 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:40:48.454696 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:40:48.454696 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:40:48.454696 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:40:48.454696 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:40:48.454696 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:40:48.454696 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:48.454696 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:48.454940 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:48.454940 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:40:48.454940 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:48.454940 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:48.454940 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:48.454940 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:40:48.454940 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:48.454940 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:40:48.454940 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:40:48.454940 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:48.454940 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:40:48.454940 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:48.454940 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:40:48.454940 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:40:48.455190 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:48.455190 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:40:48.455190 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:48.455190 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:40:48.455729 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:48.455729 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:40:48.455729 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:40:48.455729 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:40:48.455729 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:40:48.455729 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:40:48.455729 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:48.455729 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:40:48.455729 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:40:48.455729 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:48.455729 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:40:48.455729 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:40:48.455729 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:48.455729 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:48.455729 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:40:48.455729 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:48.455729 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:40:48.456110 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:40:48.456110 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:40:48.456110 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:40:48.456110 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:48.456110 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:48.456110 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:48.456110 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:40:48.456110 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:48.456110 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:48.456110 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:48.456110 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:40:48.456110 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:48.456110 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:40:48.456110 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:40:48.456110 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:48.456110 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:40:48.456110 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:48.456110 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:40:48.456110 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:40:48.456559 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:48.456559 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:40:48.456559 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:48.456559 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:40:48.456559 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:40:48.456669 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:40:48.456787 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:40:48.456925 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:40:48.457335 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:40:48.457713 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:40:48.457886 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:48.458579 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:40:48.458579 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:48.458579 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:40:48.458579 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:40:48.458579 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:48.458579 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:40:48.458579 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:48.458579 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:40:48.459451 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:48.459451 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:40:48.459451 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:48.459451 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:40:48.459451 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:40:48.459451 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:40:48.459451 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:40:48.459451 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:48.459451 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:48.459451 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:48.459451 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:40:48.459451 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:48.459809 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:48.459809 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:48.459809 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:40:48.459809 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:48.459809 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:40:48.459809 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:40:48.459809 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:48.459809 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:40:48.459809 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:48.459809 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:40:48.459809 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:40:48.459809 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:48.459809 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:40:48.460147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:48.460147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:40:48.460373 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:48.460373 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:40:48.460373 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:40:48.460373 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:40:48.460373 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:40:48.460373 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:40:48.460373 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:48.460373 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:40:48.460373 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:40:48.460373 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:48.460373 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:40:48.460373 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:40:48.460373 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:48.460373 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:48.460373 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:40:48.460373 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:48.460847 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:40:48.460847 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:40:48.460847 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:48.460847 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:48.460847 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:48.460847 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:40:48.460847 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:48.460847 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:48.460847 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:48.460847 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:40:48.460847 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:48.460847 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:40:48.460847 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:40:48.460847 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:48.460847 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:40:48.460847 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:48.460847 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:40:48.460847 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:40:48.460847 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:48.461373 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:40:48.461373 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:48.461373 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:40:52.973790 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:40:52.974663 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:40:52.974663 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:40:52.975737 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:40:53.007559 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:40:53.007975 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:53.008972 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:40:53.008972 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:53.008972 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:40:53.008972 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:40:53.008972 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:53.008972 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:40:53.008972 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:53.008972 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:40:53.010019 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.034 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:53.010019 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:40:53.010019 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:53.010019 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:40:53.010019 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:40:53.010019 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:40:53.010019 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:40:53.010019 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:53.010019 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:53.010019 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:53.010019 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:40:53.010019 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:53.010926 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:53.010926 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:53.010926 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:40:53.010926 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:53.010926 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:40:53.010926 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:40:53.010926 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:53.010926 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:40:53.010926 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:53.010926 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:40:53.010926 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:40:53.010926 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:53.010926 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:40:53.011618 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:53.011618 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:40:53.011618 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:53.011618 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:40:53.011618 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:40:53.011618 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:40:53.011618 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:40:53.011618 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:40:53.011618 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:53.011618 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:40:53.011618 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:40:53.011618 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:53.011618 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:40:53.011618 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:40:53.011618 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:53.012351 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:53.012351 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:40:53.012351 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:53.012351 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:40:53.012351 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:40:53.012351 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:53.012351 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:53.012351 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:53.012351 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:40:53.012351 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:53.012351 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:53.012351 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:53.012351 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:40:53.012351 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:53.012351 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:40:53.012351 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:40:53.012351 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:53.012351 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:40:53.012351 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:53.013307 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:40:53.013307 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:40:53.013307 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:53.013307 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:40:53.013307 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:53.013307 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:40:53.438437 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:40:53.439036 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:40:53.439237 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:40:53.440654 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:40:53.441076 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:53.442158 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:40:53.442158 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:53.442158 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:40:53.442158 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:40:53.442158 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:53.442158 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:40:53.442158 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:53.442158 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:40:53.443826 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:53.443826 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:40:53.443826 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:53.443826 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:40:53.443826 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:40:53.443826 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:40:53.443826 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:40:53.443826 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:53.443826 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:53.443826 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:53.443826 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:40:53.443826 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:53.444565 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:53.444565 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:53.444565 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:40:53.444565 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:53.444565 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:40:53.444565 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:40:53.444565 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:53.444565 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:40:53.444565 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:53.444565 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:40:53.444565 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:40:53.444565 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:53.444565 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:40:53.445266 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:53.445266 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:40:53.445585 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:53.445585 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:40:53.445585 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:40:53.445585 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:40:53.445585 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:40:53.445585 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:40:53.445585 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:53.445585 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:40:53.445585 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:40:53.445585 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:53.445585 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:40:53.445585 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:40:53.445585 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:53.445585 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:53.445585 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:40:53.445585 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:53.446349 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:40:53.446349 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:40:53.446349 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:53.446349 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:53.446349 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:53.446349 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:40:53.446349 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:53.446349 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:53.446349 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:53.446349 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:40:53.446349 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:53.446349 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:40:53.446349 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:40:53.446349 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:53.446349 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:40:53.446349 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:53.446349 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:40:53.446349 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:40:53.446349 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:53.447960 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:40:53.447960 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:53.447960 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:40:53.447960 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:40:53.447960 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:40:53.447960 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:40:53.447960 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:40:53.447960 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:40:53.448751 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:53.448751 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:40:53.448751 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:53.448751 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:40:53.448751 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:40:53.448751 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:53.448751 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:40:53.448751 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:53.448751 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:40:53.448751 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:53.448751 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:40:53.448751 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:40:53.448751 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:40:53.448751 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:53.448751 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:40:53.448751 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:53.448751 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:40:53.451028 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:53.451028 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:40:53.451028 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:53.451028 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:40:53.451028 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:53.451028 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:40:53.451028 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:40:53.451028 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:53.451028 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:40:53.451028 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:53.451028 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:40:53.451028 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:40:53.451028 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:53.451028 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:40:53.451028 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:53.451028 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:40:53.452826 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:53.452826 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:40:53.452826 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:53.452826 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:40:53.452826 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:40:53.452826 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:40:53.452826 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:40:53.452826 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:40:53.452826 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:40:53.452826 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:40:53.452826 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:53.452826 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:53.453767 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:53.453767 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:40:53.453767 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:53.453767 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:53.453767 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:53.453767 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:40:53.453767 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:53.453767 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:40:53.453767 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:40:53.453767 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:53.453767 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:40:53.453767 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:53.453767 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:40:53.453767 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:40:53.454053 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:53.454053 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:40:53.454053 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:53.454053 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:40:53.454053 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:53.454053 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:40:53.454053 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:40:53.454053 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:40:53.454053 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:40:53.454053 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:40:53.454053 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:53.454053 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:40:53.454053 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:40:53.454053 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:53.454338 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:40:53.454338 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:40:53.454338 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:53.454338 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:53.454338 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:40:53.454338 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:53.454338 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:40:53.454338 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:40:53.454338 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:40:53.454338 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:40:53.454338 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:40:53.454338 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:53.454338 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:53.454338 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:53.454338 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:40:53.454338 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:53.454338 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:53.454338 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:53.454338 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:40:53.455385 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:53.455385 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:40:53.455385 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:40:53.455385 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:53.455385 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:40:53.455385 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:53.455385 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:40:53.455385 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:40:53.455385 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:53.455385 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:40:53.455385 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:53.455385 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:40:53.455385 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:40:53.455385 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:40:53.455673 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:40:53.455673 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:40:53.455673 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:40:53.455673 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:53.455673 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:40:53.455673 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:53.455673 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:40:53.455673 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:40:53.455673 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:53.455673 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:40:53.455673 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:53.455673 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:40:53.455935 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:53.455935 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:40:53.455935 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:40:53.455935 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:40:53.455935 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:53.455935 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:40:53.455935 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:53.455935 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:40:53.455935 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:53.455935 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:40:53.455935 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:53.455935 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:40:53.455935 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:53.455935 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:40:53.455935 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:40:53.456250 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:53.456250 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:40:53.456250 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:53.456250 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:40:53.456250 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:40:53.456250 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:53.456250 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:40:53.456250 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:53.456250 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:40:53.456250 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:53.456250 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:40:53.456250 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:53.456250 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:40:53.456250 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:40:53.456644 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:40:53.456644 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:40:53.456644 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:40:53.456644 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:40:53.456644 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:53.456644 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:53.456644 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:53.456644 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:40:53.456644 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:53.456644 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:53.456644 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:53.456644 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:40:53.456644 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:53.457040 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:40:53.457040 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:40:53.457040 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:53.457040 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:40:53.457040 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:53.457040 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:40:53.457040 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:40:53.457040 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:53.457040 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:40:53.457040 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:53.457040 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:40:53.457040 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:53.457280 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:40:53.457280 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:40:53.457280 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:40:53.457280 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:40:53.457280 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:40:53.457280 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:53.457280 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:40:53.457280 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:40:53.457280 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:53.457280 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:40:53.457280 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:40:53.457280 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:53.457280 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:53.457280 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:40:53.457280 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:53.457280 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:40:53.457280 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:40:53.457280 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:40:53.457280 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:40:53.457709 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:53.457709 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:53.457709 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:53.457709 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:40:53.457709 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:53.457709 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:53.457709 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:53.457709 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:40:53.457709 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:53.457709 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:40:53.457709 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:40:53.457709 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:53.457709 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:40:53.457709 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:53.457709 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:40:53.457709 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:40:53.457709 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:53.457709 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:40:53.457709 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:53.457709 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:40:53.458090 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:40:53.458090 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:40:53.458090 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:40:53.458090 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:40:53.458090 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:40:53.458090 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:40:53.458242 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:53.458746 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:40:53.458746 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:53.458746 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:40:53.458746 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:40:53.458746 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:53.458746 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:40:53.458746 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:53.458746 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:40:53.459631 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:53.459631 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:40:53.459631 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:53.459631 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:40:53.459631 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:40:53.459631 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:40:53.459631 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:40:53.459631 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:53.459631 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:53.459631 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:53.459631 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:40:53.459631 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:53.459901 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:53.459901 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:53.459901 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:40:53.459901 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:53.459901 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:40:53.459901 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:40:53.459901 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:53.459901 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:40:53.459901 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:53.459901 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:40:53.459901 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:40:53.459901 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:53.459901 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:40:53.460147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:53.460147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:40:53.460722 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:53.460722 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:40:53.460722 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:40:53.460722 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:40:53.460722 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:40:53.460722 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:40:53.460722 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:53.460722 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:40:53.460722 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:40:53.460722 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:53.460722 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:40:53.460722 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:40:53.460722 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:53.460722 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:53.460722 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:40:53.460722 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:53.461058 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:40:53.461058 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:40:53.461058 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:53.461058 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:53.461058 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:53.461058 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:40:53.461058 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:53.461058 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:53.461058 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:53.461058 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:40:53.461058 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:53.461058 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:40:53.461058 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:40:53.461058 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:53.461058 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:40:53.461058 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:53.461058 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:40:53.461058 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:40:53.461058 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:53.461473 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:40:53.461473 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:53.461473 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:40:57.974496 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:40:57.975905 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:40:57.975905 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:40:57.977017 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:40:58.011353 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:40:58.011960 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:58.013040 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:40:58.013040 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:58.013040 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:40:58.013040 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:40:58.013040 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:58.013040 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:40:58.013040 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:58.013040 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:40:58.014012 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.037 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:58.014012 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:40:58.014012 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:58.014012 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:40:58.014012 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:40:58.014012 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:40:58.014012 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:40:58.014012 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:58.014012 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:58.014012 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:58.014012 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:40:58.014012 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:58.014446 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:58.014446 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:58.014446 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:40:58.014446 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:58.014446 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:40:58.014446 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:40:58.014446 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:58.014446 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:40:58.014446 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:58.014446 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:40:58.014446 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:40:58.014446 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:58.014446 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:40:58.014980 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:58.014980 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:40:58.015349 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:58.015349 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:40:58.015349 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:40:58.015349 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:40:58.015349 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:40:58.015349 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:40:58.015349 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:58.015349 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:40:58.015349 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:40:58.015349 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:58.015349 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:40:58.015349 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:40:58.015349 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:58.015349 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:58.015349 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:40:58.015349 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:58.015887 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:40:58.015887 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:40:58.015887 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:58.015887 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:58.015887 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:58.015887 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:40:58.015887 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:58.015887 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:58.015887 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:58.015887 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:40:58.015887 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:58.015887 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:40:58.015887 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:40:58.015887 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:58.015887 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:40:58.015887 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:58.015887 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:40:58.015887 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:40:58.015887 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:58.016465 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:40:58.016465 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:58.016465 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:40:58.439097 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:40:58.439648 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:40:58.439935 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:40:58.441183 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:40:58.441614 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:58.442636 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:40:58.442636 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:58.442636 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:40:58.442636 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:40:58.442636 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:58.442636 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:40:58.442636 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:58.442636 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:40:58.443834 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:58.443834 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:40:58.443834 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:58.443834 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:40:58.443834 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:40:58.443834 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:40:58.443834 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:40:58.443834 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:58.443834 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:58.443834 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:58.443834 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:40:58.443834 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:58.444509 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:58.444509 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:58.444509 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:40:58.444509 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:58.444509 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:40:58.444509 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:40:58.444509 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:58.444509 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:40:58.444509 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:58.444509 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:40:58.444509 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:40:58.444509 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:58.444509 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:40:58.445050 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:58.445050 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:40:58.445050 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:58.445050 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:40:58.445050 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:40:58.445050 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:40:58.445050 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:40:58.445050 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:40:58.445050 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:58.445050 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:40:58.445050 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:40:58.445050 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:58.445050 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:40:58.445050 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:40:58.445050 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:58.445767 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:58.445767 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:40:58.445767 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:58.445767 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:40:58.445767 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:40:58.445767 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:58.445767 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:58.445767 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:58.445767 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:40:58.445767 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:58.445767 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:58.445767 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:58.445767 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:40:58.445767 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:58.445767 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:40:58.445767 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:40:58.445767 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:58.445767 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:40:58.445767 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:58.446632 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:40:58.446632 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:40:58.446632 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:58.446632 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:40:58.446632 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:58.446632 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:40:58.446632 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:40:58.446855 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:40:58.446855 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:40:58.446928 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:40:58.447336 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:40:58.447566 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:58.448253 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:40:58.448253 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:58.448253 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:40:58.448253 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:40:58.448253 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:58.448253 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:40:58.448253 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:58.448253 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:40:58.448974 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:58.448974 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:40:58.448974 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:40:58.448974 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:40:58.448974 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:58.448974 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:40:58.448974 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:58.448974 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:40:58.448974 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:58.448974 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:40:58.448974 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:58.448974 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:40:58.448974 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:58.448974 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:40:58.448974 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:40:58.449613 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:58.449613 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:40:58.449613 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:58.449613 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:40:58.449613 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:40:58.449613 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:58.449613 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:40:58.449613 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:58.449613 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:40:58.449886 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:58.449886 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:40:58.449886 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:58.449886 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:40:58.449886 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:40:58.449886 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:40:58.449886 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:40:58.449886 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:40:58.449886 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:40:58.449886 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:40:58.449886 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:58.449886 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:58.450293 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:58.450293 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:40:58.450293 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:58.450293 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:58.450293 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:58.450293 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:40:58.450293 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:58.450293 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:40:58.450293 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:40:58.450293 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:58.450293 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:40:58.450293 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:58.450293 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:40:58.450293 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:40:58.450712 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:58.450712 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:40:58.450712 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:58.450712 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:40:58.451086 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:58.451086 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:40:58.451086 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:40:58.451086 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:40:58.451086 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:40:58.451086 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:40:58.451086 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:58.451086 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:40:58.451086 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:40:58.451086 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:58.451086 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:40:58.451086 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:40:58.451086 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:58.451086 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:58.451086 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:40:58.451086 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:58.451580 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:40:58.451580 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:40:58.451580 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:40:58.451580 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:40:58.451580 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:40:58.451580 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:58.451580 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:58.451580 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:58.451580 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:40:58.451580 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:58.451580 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:58.451580 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:58.451580 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:40:58.451580 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:58.451580 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:40:58.451580 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:40:58.451580 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:58.451580 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:40:58.452162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:58.452162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:40:58.452162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:40:58.452162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:58.452162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:40:58.452162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:58.452162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:40:58.452162 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:40:58.452162 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:40:58.452521 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:40:58.452521 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:40:58.452913 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:40:58.453204 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:58.453820 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:40:58.453820 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:58.453820 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:40:58.453820 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:40:58.453820 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:58.453820 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:40:58.453820 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:58.453820 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:40:58.454694 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:58.454694 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:40:58.454694 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:40:58.454694 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:40:58.454694 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:58.454694 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:40:58.454694 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:58.454694 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:40:58.454694 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:58.454694 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:40:58.454694 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:58.454694 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:40:58.454694 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:58.454694 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:40:58.454694 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:40:58.455137 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:58.455137 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:40:58.455137 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:58.455137 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:40:58.455137 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:40:58.455137 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:58.455137 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:40:58.455137 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:58.455137 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:40:58.455615 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:58.455615 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:40:58.455615 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:58.455615 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:40:58.455615 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:40:58.455615 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:40:58.455615 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:40:58.455615 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:40:58.455615 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:40:58.455615 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:58.455615 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:58.456040 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:58.456040 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:40:58.456040 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:58.456040 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:58.456040 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:58.456040 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:40:58.456040 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:58.456040 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:40:58.456040 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:40:58.456040 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:58.456040 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:40:58.456040 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:58.456040 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:40:58.456040 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:40:58.456471 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:58.456471 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:40:58.456471 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:58.456471 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:40:58.456782 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:58.456782 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:40:58.456782 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:40:58.456782 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:40:58.456782 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:40:58.456782 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:40:58.456782 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:58.456782 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:40:58.456782 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:40:58.456782 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:58.456782 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:40:58.456782 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:40:58.456782 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:58.456782 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:58.456782 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:40:58.456782 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:58.456782 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:40:58.457313 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:40:58.457313 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:40:58.457313 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:40:58.457313 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:58.457313 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:58.457313 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:58.457313 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:40:58.457313 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:58.457313 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:58.457313 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:58.457313 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:40:58.457313 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:58.457313 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:40:58.457313 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:40:58.457313 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:58.457313 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:40:58.457313 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:58.457313 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:40:58.457313 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:40:58.457829 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:58.457829 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:40:58.457829 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:58.457829 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:40:58.457829 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:40:58.457829 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:40:58.457829 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:40:58.458053 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:40:58.458309 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:40:58.458552 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:40:58.458665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:58.459220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:40:58.459220 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:58.459220 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:40:58.459220 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:40:58.459220 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:58.459220 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:40:58.459220 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:58.459220 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:40:58.460126 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:58.460126 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:40:58.460126 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:58.460126 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:40:58.460126 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:40:58.460126 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:40:58.460126 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:40:58.460126 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:58.460126 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:58.460126 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:58.460126 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:40:58.460126 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:58.460502 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:40:58.460502 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:58.460502 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:40:58.460502 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:58.460502 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:40:58.460502 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:40:58.460502 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:58.460502 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:40:58.460502 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:58.460502 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:40:58.460502 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:40:58.460502 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:58.460502 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:40:58.460865 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:58.460865 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:40:58.461214 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:58.461214 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:40:58.461214 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:40:58.461214 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:40:58.461214 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:40:58.461214 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:40:58.461214 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:58.461214 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:40:58.461214 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:40:58.461214 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:58.461214 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:40:58.461214 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:40:58.461214 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:40:58.461214 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:40:58.461214 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:40:58.461214 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:40:58.462190 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:40:58.462190 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:40:58.462190 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:58.462190 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:58.462190 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:40:58.462190 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:40:58.462190 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:40:58.462190 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:40:58.462190 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:40:58.462190 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:40:58.462190 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:40:58.462190 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:40:58.462190 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:40:58.462190 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:40:58.462190 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:40:58.462190 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:40:58.462190 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:40:58.462190 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:40:58.462190 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:40:58.463137 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:40:58.463137 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:40:58.463137 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:41:02.975129 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:41:02.975738 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:41:02.975866 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:41:02.976938 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:41:03.014974 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:41:03.015507 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:03.016714 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:41:03.016714 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:03.016714 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:41:03.016714 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:41:03.016714 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:03.016714 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:41:03.016714 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:03.016714 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:41:03.017838 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.041 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:03.017838 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:41:03.017838 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:03.017838 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:41:03.017838 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:41:03.017838 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:41:03.017838 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:41:03.017838 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:03.017838 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:03.017838 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:03.017838 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:41:03.017838 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:03.018649 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:03.018649 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:03.018649 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:41:03.018649 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:03.018649 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:41:03.018649 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:41:03.018649 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:03.018649 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:41:03.018649 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:03.018649 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:41:03.018649 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:41:03.018649 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:03.018649 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:41:03.019393 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:03.019393 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:41:03.019393 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:03.019393 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:41:03.019393 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:41:03.019393 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:41:03.019393 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:41:03.019393 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:41:03.019393 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:03.019393 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:41:03.019393 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:41:03.019393 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:03.019393 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:41:03.019393 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:41:03.019393 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:03.020304 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:03.020304 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:41:03.020304 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:03.020304 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:41:03.020304 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:41:03.020304 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:03.020304 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:03.020304 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:03.020304 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:41:03.020304 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:03.020304 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:03.020304 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:03.020304 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:41:03.020304 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:03.020304 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:41:03.020304 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:41:03.020304 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:03.020304 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:41:03.020304 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:03.021368 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:41:03.021368 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:41:03.021368 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:03.021368 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:41:03.021368 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:03.021368 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:41:03.439756 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:41:03.440478 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:41:03.440699 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:41:03.441872 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:41:03.442185 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:03.443184 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:41:03.443184 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:03.443184 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:41:03.443184 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:41:03.443184 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:03.443184 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:41:03.443184 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:03.443184 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:41:03.445235 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:03.445235 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:41:03.445235 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:03.445235 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:41:03.445235 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:41:03.445235 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:41:03.445235 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:41:03.445235 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:03.445235 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:03.445235 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:03.445235 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:41:03.445235 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:03.445900 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:03.445900 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:03.445900 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:41:03.445900 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:03.445900 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:41:03.445900 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:41:03.445900 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:03.445900 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:41:03.445900 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:03.445900 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:41:03.445900 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:41:03.445900 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:03.445900 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:41:03.446456 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:03.446456 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:41:03.447100 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:03.447100 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:41:03.447100 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:41:03.447100 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:41:03.447100 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:41:03.447100 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:41:03.447100 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:03.447100 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:41:03.447100 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:41:03.447100 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:03.447100 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:41:03.447100 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:41:03.447100 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:03.447100 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:03.447100 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:41:03.447100 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:03.448593 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:41:03.448593 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:41:03.448593 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:03.448593 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:03.448593 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:03.448593 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:41:03.448593 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:03.448593 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:03.448593 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:03.448593 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:41:03.448593 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:03.448593 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:41:03.448593 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:41:03.448593 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:03.448593 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:41:03.448593 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:03.448593 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:41:03.448593 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:41:03.448593 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:03.450124 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:41:03.450124 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:03.450124 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:41:03.450124 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:41:03.450124 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:41:03.450124 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:41:03.450124 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:41:03.450124 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:41:03.450552 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:03.450552 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:41:03.450552 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:03.450552 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:41:03.450552 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:41:03.450552 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:03.450552 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:41:03.450552 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:03.450552 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:41:03.450902 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:03.450902 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:41:03.450902 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:41:03.450902 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:41:03.450902 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:03.450902 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:41:03.450902 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:03.450902 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:41:03.450902 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:03.450902 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:41:03.450902 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:03.450902 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:41:03.450902 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:03.450902 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:41:03.450902 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:41:03.452118 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:03.452118 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:41:03.452118 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:03.452118 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:41:03.452118 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:41:03.452118 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:03.452118 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:41:03.452118 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:03.452118 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:41:03.452864 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:03.452864 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:41:03.452864 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:03.452864 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:41:03.452864 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:41:03.452864 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:41:03.452864 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:41:03.452864 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:41:03.452864 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:41:03.452864 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:41:03.452864 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:03.452864 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:03.453488 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:03.453488 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:41:03.453488 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:03.453488 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:03.453488 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:03.453488 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:41:03.453488 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:03.453488 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:41:03.453488 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:41:03.453488 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:03.453488 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:41:03.453488 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:03.453488 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:41:03.453488 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:41:03.454065 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:03.454065 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:41:03.454065 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:03.454065 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:41:03.454065 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:03.454065 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:41:03.454065 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:41:03.454065 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:41:03.454065 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:41:03.454065 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:41:03.454065 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:03.454065 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:41:03.454065 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:41:03.454065 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:03.454432 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:41:03.454432 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:41:03.454432 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:03.454432 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:03.454432 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:41:03.454432 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:03.454432 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:41:03.454432 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:41:03.454432 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:41:03.454432 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:41:03.454432 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:41:03.454432 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:03.454432 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:03.454432 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:03.454432 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:41:03.454432 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:03.454432 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:03.454432 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:03.454432 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:41:03.454896 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:03.454896 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:41:03.454896 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:41:03.454896 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:03.454896 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:41:03.454896 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:03.454896 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:41:03.454896 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:41:03.454896 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:03.454896 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:41:03.454896 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:03.454896 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:41:03.454896 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:41:03.454896 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:41:03.455243 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:41:03.455243 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:41:03.455243 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:41:03.455243 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:03.455864 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:41:03.455864 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:03.455864 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:41:03.455864 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:41:03.455864 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:03.455864 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:41:03.455864 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:03.455864 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:41:03.456281 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:03.456281 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:41:03.456281 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:41:03.456281 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:41:03.456281 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:03.456281 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:41:03.456281 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:03.456281 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:41:03.456281 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:03.456281 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:41:03.456281 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:03.456281 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:41:03.456281 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:03.456281 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:41:03.456281 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:41:03.457031 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:03.457031 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:41:03.457031 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:03.457031 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:41:03.457031 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:41:03.457031 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:03.457031 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:41:03.457031 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:03.457031 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:41:03.457489 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:03.457489 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:41:03.457489 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:03.457489 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:41:03.457489 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:41:03.457489 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:41:03.457489 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:41:03.457489 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:41:03.457489 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:41:03.457489 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:03.457489 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:03.458067 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:03.458067 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:41:03.458067 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:03.458067 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:03.458067 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:03.458067 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:41:03.458067 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:03.458067 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:41:03.458067 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:41:03.458067 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:03.458067 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:41:03.458067 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:03.458067 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:41:03.458067 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:41:03.458604 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:03.458604 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:41:03.458604 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:03.458604 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:41:03.458604 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:03.458604 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:41:03.458604 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:41:03.458604 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:41:03.458604 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:41:03.458604 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:41:03.458604 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:03.458604 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:41:03.458604 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:41:03.458604 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:03.458951 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:41:03.458951 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:41:03.458951 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:03.458951 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:03.458951 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:41:03.458951 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:03.458951 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:41:03.458951 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:41:03.458951 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:41:03.458951 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:41:03.458951 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:03.458951 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:03.458951 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:03.458951 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:41:03.458951 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:03.458951 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:03.458951 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:03.458951 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:41:03.458951 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:03.459412 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:41:03.459412 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:41:03.459412 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:03.459412 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:41:03.459412 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:03.459412 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:41:03.459412 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:41:03.459412 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:03.459412 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:41:03.459412 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:03.459412 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:41:03.459412 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:41:03.459412 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:41:03.459749 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:41:03.459749 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:41:03.459749 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:41:03.459749 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:41:03.459749 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:03.460419 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:41:03.460419 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:03.460419 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:41:03.460419 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:41:03.460419 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:03.460419 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:41:03.460419 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:03.460419 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:41:03.461156 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:03.461156 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:41:03.461156 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:03.461156 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:41:03.461156 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:41:03.461156 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:41:03.461156 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:41:03.461156 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:03.461156 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:03.461156 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:03.461156 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:41:03.461156 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:03.461783 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:03.461783 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:03.461783 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:41:03.461783 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:03.461783 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:41:03.461783 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:41:03.461783 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:03.461783 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:41:03.461783 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:03.461783 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:41:03.461783 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:41:03.461783 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:03.461783 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:41:03.462302 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:03.462302 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:41:03.462302 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:03.462302 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:41:03.462302 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:41:03.462302 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:41:03.462302 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:41:03.462302 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:41:03.462302 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:03.462302 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:41:03.462302 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:41:03.462302 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:03.462302 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:41:03.462302 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:41:03.462302 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:03.462861 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:03.462861 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:41:03.462861 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:03.462861 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:41:03.462861 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:41:03.462861 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:03.462861 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:03.462861 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:03.462861 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:41:03.462861 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:03.462861 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:03.462861 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:03.462861 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:41:03.462861 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:03.462861 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:41:03.462861 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:41:03.462861 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:03.462861 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:41:03.462861 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:03.463613 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:41:03.463613 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:41:03.463613 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:03.463613 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:41:03.463613 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:03.463613 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:41:07.975817 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:41:07.976424 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:41:07.976487 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:41:07.977296 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:41:08.013606 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:41:08.014510 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:08.014882 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:41:08.014882 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:08.014882 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:41:08.014882 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:41:08.014882 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:08.014882 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:41:08.014882 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:08.014882 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:41:08.015702 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.038 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:08.015702 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:41:08.015702 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:08.015702 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:41:08.015702 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:41:08.015702 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:41:08.015702 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:41:08.015702 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:08.015702 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:08.015702 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:08.015702 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:41:08.015702 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:08.016432 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:08.016432 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:08.016432 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:41:08.016432 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:08.016432 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:41:08.016432 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:41:08.016432 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:08.016432 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:41:08.016432 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:08.016432 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:41:08.016432 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:41:08.016432 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:08.016432 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:41:08.017047 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:08.017047 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:41:08.017047 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:08.017047 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:41:08.017047 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:41:08.017047 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:41:08.017047 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:41:08.017047 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:41:08.017047 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:08.017047 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:41:08.017047 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:41:08.017047 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:08.017047 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:41:08.017047 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:41:08.017047 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:08.017762 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:08.017762 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:41:08.017762 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:08.017762 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:41:08.017762 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:41:08.017762 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:08.017762 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:08.017762 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:08.017762 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:41:08.017762 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:08.017762 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:08.017762 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:08.017762 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:41:08.017762 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:08.017762 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:41:08.017762 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:41:08.017762 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:08.017762 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:41:08.017762 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:08.018571 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:41:08.018571 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:41:08.018571 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:08.018571 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:41:08.018571 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:08.018571 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:41:08.440490 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:41:08.440978 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:41:08.440978 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:41:08.442094 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:41:08.442422 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:08.443403 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:41:08.443403 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:08.443403 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:41:08.443403 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:41:08.443403 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:08.443403 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:41:08.443403 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:08.443403 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:41:08.444936 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:08.444936 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:41:08.444936 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:08.444936 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:41:08.444936 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:41:08.444936 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:41:08.444936 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:41:08.444936 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:08.444936 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:08.444936 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:08.444936 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:41:08.444936 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:08.445497 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:08.445497 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:08.445497 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:41:08.445497 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:08.445497 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:41:08.445497 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:41:08.445497 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:08.445497 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:41:08.445497 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:08.445497 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:41:08.445497 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:41:08.445497 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:08.445497 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:41:08.446075 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:08.446075 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:41:08.447111 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:08.447111 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:41:08.447111 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:41:08.447111 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:41:08.447111 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:41:08.447111 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:41:08.447111 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:08.447111 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:41:08.447111 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:41:08.447111 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:08.447111 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:41:08.447111 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:41:08.447111 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:08.447111 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:08.447111 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:41:08.447111 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:08.448691 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:41:08.448691 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:41:08.448691 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:08.448691 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:08.448691 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:08.448691 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:41:08.448691 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:08.448691 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:08.448691 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:08.448691 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:41:08.448691 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:08.448691 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:41:08.448691 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:41:08.448691 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:08.448691 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:41:08.448691 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:08.448691 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:41:08.448691 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:41:08.448691 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:08.449774 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:41:08.449774 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:08.449774 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:41:08.449774 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:41:08.449774 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:41:08.449774 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:41:08.449774 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:41:08.449774 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:41:08.450235 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:08.450235 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:41:08.450235 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:08.450235 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:41:08.450235 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:41:08.450235 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:08.450235 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:41:08.450235 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:08.450235 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:41:08.450768 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:08.450768 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:41:08.450768 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:41:08.450768 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:41:08.450768 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:08.450768 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:41:08.450768 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:08.450768 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:41:08.450768 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:08.450768 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:41:08.450768 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:08.450768 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:41:08.450768 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:08.450768 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:41:08.450768 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:41:08.452756 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:08.452756 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:41:08.452756 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:08.452756 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:41:08.452756 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:41:08.452756 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:08.452756 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:41:08.452756 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:08.452756 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:41:08.452756 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:08.452756 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:41:08.452756 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:08.452756 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:41:08.452756 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:41:08.454085 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:41:08.454085 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:41:08.454085 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:41:08.454085 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:41:08.454085 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:41:08.454085 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:08.454085 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:08.454085 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:08.454085 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:41:08.454085 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:08.454085 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:08.454085 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:08.454085 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:41:08.454835 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:08.454835 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:41:08.454835 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:41:08.454835 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:08.454835 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:41:08.454835 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:08.454835 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:41:08.454835 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:41:08.454835 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:08.454835 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:41:08.454835 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:08.454835 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:41:08.455532 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:08.455532 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:41:08.455532 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:41:08.455532 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:41:08.455532 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:41:08.455532 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:41:08.455532 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:08.455532 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:41:08.455532 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:41:08.455532 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:08.455532 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:41:08.455532 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:41:08.455532 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:08.455532 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:08.455532 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:41:08.455532 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:08.456407 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:41:08.456407 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:41:08.456407 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:41:08.456407 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:41:08.456407 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:41:08.456407 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:08.456407 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:08.456407 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:08.456407 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:41:08.456407 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:08.456407 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:08.456407 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:08.456407 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:41:08.456407 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:08.456407 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:41:08.456407 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:41:08.456407 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:08.456407 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:41:08.457635 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:08.457635 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:41:08.457635 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:41:08.457635 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:08.457635 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:41:08.457635 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:08.457635 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:41:08.457635 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:41:08.457635 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:41:08.457635 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:41:08.457998 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:41:08.457998 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:41:08.457998 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:08.458568 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:41:08.458568 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:08.458568 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:41:08.458568 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:41:08.458568 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:08.458568 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:41:08.458568 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:08.458568 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:41:08.459410 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:08.459410 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:41:08.459410 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:41:08.459410 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:41:08.459410 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:08.459410 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:41:08.459410 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:08.459410 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:41:08.459410 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:08.459410 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:41:08.459410 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:08.459410 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:41:08.459410 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:08.459410 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:41:08.459410 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:41:08.459937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:08.459937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:41:08.459937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:08.459937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:41:08.459937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:41:08.459937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:08.459937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:41:08.459937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:08.459937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:41:08.459937 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:41:08.461083 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:08.461083 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:41:08.461083 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:08.461083 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:41:08.461083 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:41:08.461083 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:41:08.461083 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:41:08.461083 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:41:08.461083 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:41:08.461083 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:08.461083 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:08.461549 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:08.461549 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:41:08.461549 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:08.461549 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:08.461549 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:08.461549 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:41:08.461549 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:08.461549 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:41:08.461549 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:41:08.461549 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:08.461549 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:41:08.461549 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:08.461549 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:41:08.461549 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:41:08.462090 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:08.462090 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:41:08.462090 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:08.462090 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:41:08.462433 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:08.462433 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:41:08.462433 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:41:08.462433 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:41:08.462433 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:41:08.462433 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:41:08.462433 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:08.462433 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:41:08.462433 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:41:08.462433 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:08.462433 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:41:08.462433 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:41:08.462433 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:08.462433 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:08.462433 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:41:08.462433 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:08.462433 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:41:08.463464 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:41:08.463464 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:41:08.463464 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:41:08.463464 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:08.463464 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:08.463464 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:08.463464 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:41:08.463464 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:08.463464 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:08.463464 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:08.463464 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:41:08.463464 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:08.463464 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:41:08.463464 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:41:08.463464 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:08.463464 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:41:08.463464 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:08.463464 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:41:08.463464 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:41:08.464303 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:08.464303 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:41:08.464303 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:08.464303 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:41:08.464303 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:41:08.464303 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:41:08.464303 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:41:08.464303 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:41:08.464711 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:41:08.464711 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:08.464711 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:41:08.464711 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:08.464711 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:41:08.464711 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:41:08.464711 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:08.464711 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:41:08.464711 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:08.464711 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:41:08.465500 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:08.465500 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:41:08.465500 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:08.465500 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:41:08.465500 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:41:08.465500 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:41:08.465500 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:41:08.465500 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:08.465500 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:08.465500 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:08.465500 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:41:08.465500 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:08.465787 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:08.465787 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:08.465787 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:41:08.465787 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:08.465787 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:41:08.465787 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:41:08.465787 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:08.465787 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:41:08.465787 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:08.465787 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:41:08.465787 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:41:08.465787 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:08.465787 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:41:08.466023 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:08.466023 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:41:08.466462 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:08.466462 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:41:08.466462 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:41:08.466462 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:41:08.466462 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:41:08.466462 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:41:08.466462 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:08.466462 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:41:08.466462 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:41:08.466462 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:08.466462 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:41:08.466462 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:41:08.466462 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:08.466462 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:08.466462 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:41:08.466462 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:08.466788 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:41:08.466788 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:41:08.466788 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:08.466788 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:08.466788 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:08.466788 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:41:08.466788 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:08.466788 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:08.466788 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:08.466788 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:41:08.466788 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:08.466788 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:41:08.466788 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:41:08.466788 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:08.466788 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:41:08.466788 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:08.466788 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:41:08.466788 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:41:08.466788 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:08.467134 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:41:08.467134 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:08.467134 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:41:12.976520 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:41:12.977943 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:41:12.978137 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:41:12.979458 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:41:13.007857 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:41:13.008237 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:13.009116 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:41:13.009116 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:13.009116 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:41:13.009116 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:41:13.009116 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:13.009116 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:41:13.009116 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:13.009116 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:41:13.009906 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.031 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:13.009906 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:41:13.009906 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:13.009906 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:41:13.009906 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:41:13.009906 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:41:13.009906 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:41:13.009906 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:13.009906 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:13.009906 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:13.009906 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:41:13.009906 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:13.010555 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:13.010555 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:13.010555 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:41:13.010555 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:13.010555 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:41:13.010555 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:41:13.010555 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:13.010555 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:41:13.010555 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:13.010555 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:41:13.010555 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:41:13.010555 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:13.010555 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:41:13.011069 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:13.011069 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:41:13.011069 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:13.011069 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:41:13.011069 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:41:13.011069 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:41:13.011069 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:41:13.011069 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:41:13.011069 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:13.011069 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:41:13.011069 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:41:13.011069 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:13.011069 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:41:13.011069 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:41:13.011069 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:13.011863 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:13.011863 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:41:13.011863 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:13.011863 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:41:13.011863 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:41:13.011863 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:13.011863 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:13.011863 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:13.011863 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:41:13.011863 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:13.011863 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:13.011863 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:13.011863 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:41:13.011863 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:13.011863 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:41:13.011863 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:41:13.011863 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:13.011863 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:41:13.011863 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:13.012767 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:41:13.012767 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:41:13.012767 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:13.012767 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:41:13.012767 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:13.012767 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:41:13.441124 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:41:13.441866 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:41:13.442083 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:41:13.443722 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:41:13.444170 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:13.445362 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:41:13.445362 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:13.445362 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:41:13.445362 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:41:13.445362 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:13.445362 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:41:13.445362 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:13.445362 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:41:13.447014 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:13.447014 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:41:13.447014 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:13.447014 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:41:13.447014 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:41:13.447014 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:41:13.447014 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:41:13.447014 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:13.447014 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:13.447014 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:13.447014 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:41:13.447014 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:13.447647 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:13.447647 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:13.447647 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:41:13.447647 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:13.447647 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:41:13.447647 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:41:13.447647 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:13.447647 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:41:13.447647 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:13.447647 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:41:13.447647 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:41:13.447647 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:13.447647 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:41:13.448139 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:13.448139 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:41:13.449592 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:13.449592 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:41:13.449592 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:41:13.449592 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:41:13.449592 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:41:13.449592 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:41:13.449592 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:13.449592 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:41:13.449592 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:41:13.449592 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:13.449592 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:41:13.449592 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:41:13.449592 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:13.449592 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:13.449592 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:41:13.449592 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:13.450288 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:41:13.450288 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:41:13.450288 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:13.450288 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:13.450288 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:13.450288 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:41:13.450288 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:13.450288 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:13.450288 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:13.450288 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:41:13.450288 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:13.450288 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:41:13.450288 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:41:13.450288 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:13.450288 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:41:13.450288 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:13.450288 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:41:13.450288 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:41:13.450288 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:13.451072 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:41:13.451072 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:13.451072 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:41:13.451072 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:41:13.451072 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:41:13.451072 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:41:13.451072 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:41:13.451442 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:41:13.451946 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:13.452870 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:41:13.452870 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:13.452870 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:41:13.452870 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:41:13.452870 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:13.452870 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:41:13.452870 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:13.452870 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:41:13.454120 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:13.454120 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:41:13.454120 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:41:13.454120 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:41:13.454120 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:13.454120 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:41:13.454120 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:13.454120 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:41:13.454120 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:13.454120 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:41:13.454120 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:13.454120 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:41:13.454120 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:13.454120 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:41:13.454120 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:41:13.454631 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:13.454631 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:41:13.454631 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:13.454631 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:41:13.454631 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:41:13.454631 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:13.454631 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:41:13.454631 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:13.454631 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:41:13.456260 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:13.456260 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:41:13.456260 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:13.456260 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:41:13.456260 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:41:13.456260 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:41:13.456260 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:41:13.456260 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:41:13.456260 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:41:13.456260 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:41:13.456260 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:13.456260 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:13.456688 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:13.456688 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:41:13.456688 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:13.456688 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:13.456688 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:13.456688 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:41:13.456688 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:13.456688 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:41:13.456688 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:41:13.456688 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:13.456688 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:41:13.456688 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:13.456688 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:41:13.456688 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:41:13.457038 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:13.457038 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:41:13.457038 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:13.457038 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:41:13.457038 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:41:13.457567 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:13.457567 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:41:13.457567 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:41:13.457567 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:41:13.457567 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:41:13.457567 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:41:13.457567 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:13.457567 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:41:13.457567 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:41:13.457567 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:13.457567 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:41:13.457567 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:41:13.457567 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:13.457567 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:13.457567 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:41:13.457567 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:13.457959 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:41:13.457959 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:41:13.457959 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:41:13.457959 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:41:13.457959 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:41:13.457959 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:13.457959 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:13.457959 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:13.457959 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:41:13.457959 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:13.457959 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:13.457959 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:13.457959 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:41:13.457959 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:13.457959 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:41:13.457959 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:41:13.457959 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:13.457959 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:41:13.458432 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:13.458432 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:41:13.458432 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:41:13.458432 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:13.458432 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:41:13.458432 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:13.458432 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:41:13.458432 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:41:13.458432 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:41:13.458432 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:41:13.458705 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:41:13.458884 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:13.460295 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:41:13.460295 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:13.460295 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:41:13.460295 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:41:13.460295 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:13.460295 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:41:13.460295 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:13.460295 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:41:13.461028 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:13.461028 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:41:13.461028 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:41:13.461028 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:41:13.461028 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:13.461028 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:41:13.461028 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:13.461028 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:41:13.461028 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:13.461028 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:41:13.461028 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:13.461028 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:41:13.461028 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:13.461028 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:41:13.461028 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:41:13.461440 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:13.461440 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:41:13.461440 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:13.461440 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:41:13.461440 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:41:13.461440 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:13.461440 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:41:13.461440 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:13.461440 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:41:13.461734 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:13.461734 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:41:13.461734 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:13.461734 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:41:13.461734 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:41:13.461734 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:41:13.461734 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:41:13.461734 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:41:13.461734 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:41:13.461734 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:13.461734 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:13.462131 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:13.462131 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:41:13.462131 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:13.462131 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:13.462131 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:13.462131 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:41:13.462131 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:13.462131 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:41:13.462131 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:41:13.462131 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:13.462131 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:41:13.462131 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:13.462131 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:41:13.462131 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:41:13.462495 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:13.462495 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:41:13.462495 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:13.462495 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:41:13.462495 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:13.462495 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:41:13.462495 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:41:13.462495 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:41:13.462495 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:41:13.462495 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:41:13.462495 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:13.462495 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:41:13.462495 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:41:13.462495 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:13.462903 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:41:13.462903 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:41:13.462903 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:13.462903 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:13.462903 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:41:13.462903 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:13.462903 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:41:13.462903 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:41:13.462903 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:41:13.462903 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:41:13.462903 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:13.462903 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:13.462903 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:13.462903 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:41:13.462903 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:13.462903 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:13.462903 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:13.462903 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:41:13.462903 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:13.463367 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:41:13.463367 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:41:13.463367 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:13.463367 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:41:13.463367 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:13.463367 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:41:13.463367 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:41:13.463367 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:13.463367 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:41:13.463367 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:13.463367 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:41:13.463367 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:41:13.463367 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:41:13.463775 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:41:13.463775 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:41:13.464080 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:41:13.464629 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:41:13.464913 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:13.465846 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:41:13.465846 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:13.465846 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:41:13.465846 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:41:13.465846 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:13.465846 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:41:13.465846 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:13.465846 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:41:13.466717 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:13.466717 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:41:13.466717 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:13.466717 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:41:13.466717 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:41:13.466717 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:41:13.466717 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:41:13.466717 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:13.466717 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:13.466717 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:13.466717 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:41:13.466717 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:13.467047 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:13.467047 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:13.467047 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:41:13.467047 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:13.467047 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:41:13.467047 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:41:13.467047 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:13.467047 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:41:13.467047 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:13.467047 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:41:13.467047 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:41:13.467047 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:13.467047 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:41:13.467379 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:13.467379 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:41:13.467543 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:13.467543 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:41:13.467543 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:41:13.467543 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:41:13.467543 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:41:13.467543 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:41:13.467543 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:13.467543 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:41:13.467543 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:41:13.467543 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:13.467543 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:41:13.467543 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:41:13.467543 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:13.467543 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:13.467543 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:41:13.467543 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:13.467965 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:41:13.467965 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:41:13.467965 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:13.467965 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:13.467965 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:13.467965 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:41:13.467965 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:13.467965 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:13.467965 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:13.467965 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:41:13.467965 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:13.467965 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:41:13.467965 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:41:13.467965 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:13.467965 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:41:13.467965 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:13.467965 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:41:13.467965 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:41:13.467965 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:13.468458 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:41:13.468458 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:13.468458 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:41:17.977965 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:41:17.978899 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:41:17.979152 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:41:17.980931 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:41:18.016631 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:41:18.017114 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:18.018238 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:41:18.018238 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:18.018238 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:41:18.018238 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:41:18.018238 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:18.018238 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:41:18.018238 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:18.018238 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:41:18.019411 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.039 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:18.019411 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:41:18.019411 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:18.019411 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:41:18.019411 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:41:18.019411 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:41:18.019411 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:41:18.019411 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:18.019411 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:18.019411 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:18.019411 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:41:18.019411 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:18.020039 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:18.020039 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:18.020039 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:41:18.020039 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:18.020039 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:41:18.020039 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:41:18.020039 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:18.020039 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:41:18.020039 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:18.020039 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:41:18.020039 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:41:18.020039 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:18.020039 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:41:18.020704 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:18.020704 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:41:18.020704 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:18.020704 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:41:18.020704 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:41:18.020704 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:41:18.020704 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:41:18.020704 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:41:18.020704 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:18.020704 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:41:18.020704 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:41:18.020704 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:18.020704 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:41:18.020704 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:41:18.020704 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:18.021584 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:18.021584 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:41:18.021584 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:18.021584 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:41:18.021584 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:41:18.021584 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:18.021584 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:18.021584 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:18.021584 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:41:18.021584 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:18.021584 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:18.021584 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:18.021584 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:41:18.021584 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:18.021584 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:41:18.021584 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:41:18.021584 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:18.021584 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:41:18.021584 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:18.022482 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:41:18.022482 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:41:18.022482 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:18.022482 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:41:18.022482 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:18.022482 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:41:18.442053 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:41:18.442470 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:41:18.442654 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:41:18.443536 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:41:18.443779 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:18.444516 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:41:18.444516 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:18.444516 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:41:18.444516 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:41:18.444516 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:18.444516 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:41:18.444516 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:18.444516 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:41:18.446056 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:18.446056 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:41:18.446056 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:18.446056 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:41:18.446056 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:41:18.446056 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:41:18.446056 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:41:18.446056 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:18.446056 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:18.446056 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:18.446056 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:41:18.446056 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:18.446717 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:18.446717 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:18.446717 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:41:18.446717 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:18.446717 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:41:18.446717 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:41:18.446717 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:18.446717 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:41:18.446717 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:18.446717 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:41:18.446717 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:41:18.446717 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:18.446717 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:41:18.447305 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:18.447305 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:41:18.447728 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:18.447728 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:41:18.447728 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:41:18.447728 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:41:18.447728 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:41:18.447728 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:41:18.447728 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:18.447728 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:41:18.447728 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:41:18.447728 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:18.447728 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:41:18.447728 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:41:18.447728 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:18.447728 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:18.447728 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:41:18.447728 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:18.448500 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:41:18.448500 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:41:18.448500 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:18.448500 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:18.448500 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:18.448500 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:41:18.448500 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:18.448500 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:18.448500 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:18.448500 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:41:18.448500 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:18.448500 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:41:18.448500 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:41:18.448500 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:18.448500 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:41:18.448500 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:18.448500 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:41:18.448500 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:41:18.448500 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:18.449436 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:41:18.449436 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:18.449436 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:41:18.450221 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:41:18.450556 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:41:18.450745 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:41:18.450910 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:41:18.451465 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:41:18.451796 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:18.452613 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:41:18.452613 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:18.452613 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:41:18.452613 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:41:18.452613 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:18.452613 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:41:18.452613 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:18.452613 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:41:18.453480 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:18.453480 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:41:18.453480 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:41:18.453480 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:41:18.453480 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:18.453480 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:41:18.453480 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:18.453480 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:41:18.453480 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:18.453480 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:41:18.453480 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:18.453480 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:41:18.453480 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:18.453480 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:41:18.453480 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:41:18.453961 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:18.453961 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:41:18.453961 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:18.453961 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:41:18.453961 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:41:18.453961 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:18.453961 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:41:18.453961 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:18.453961 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:41:18.454378 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:18.454378 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:41:18.454378 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:18.454378 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:41:18.454378 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:41:18.454378 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:41:18.454378 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:41:18.454378 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:41:18.454378 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:41:18.454378 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:41:18.454378 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:18.454378 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:18.454727 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:18.454727 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:41:18.454727 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:18.454727 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:18.454727 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:18.454727 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:41:18.454727 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:18.454727 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:41:18.454727 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:41:18.454727 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:18.454727 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:41:18.454727 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:18.454727 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:41:18.454727 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:41:18.455081 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:18.455081 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:41:18.455081 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:18.455081 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:41:18.455444 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:18.455444 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:41:18.455444 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:41:18.455444 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:41:18.455444 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:41:18.455444 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:41:18.455444 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:18.455444 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:41:18.455444 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:41:18.455444 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:18.455444 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:41:18.455444 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:41:18.455444 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:18.455444 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:18.455444 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:41:18.455444 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:18.455920 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:41:18.455920 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:41:18.455920 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:41:18.455920 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:41:18.455920 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:41:18.455920 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:18.455920 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:18.455920 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:18.455920 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:41:18.455920 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:18.455920 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:18.455920 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:18.455920 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:41:18.455920 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:18.455920 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:41:18.455920 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:41:18.455920 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:18.455920 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:41:18.456425 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:18.456425 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:41:18.456425 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:41:18.456425 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:18.456425 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:41:18.456425 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:18.456425 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:41:18.457918 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:41:18.458208 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:41:18.458341 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:41:18.458481 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:41:18.458953 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:41:18.459342 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:18.460054 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:41:18.460054 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:18.460054 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:41:18.460054 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:41:18.460054 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:18.460054 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:41:18.460054 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:18.460054 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:41:18.460865 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:18.460865 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:41:18.460865 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:41:18.460865 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:41:18.460865 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:18.460865 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:41:18.460865 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:18.460865 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:41:18.460865 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:18.460865 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:41:18.460865 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:18.460865 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:41:18.460865 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:18.460865 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:41:18.460865 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:41:18.461303 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:18.461303 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:41:18.461303 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:18.461303 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:41:18.461303 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:41:18.461303 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:18.461303 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:41:18.461303 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:18.461303 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:41:18.461862 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:18.461862 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:41:18.461862 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:18.461862 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:41:18.461862 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:41:18.461862 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:41:18.461862 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:41:18.461862 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:41:18.461862 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:41:18.461862 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:18.461862 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:18.462260 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:18.462260 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:41:18.462260 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:18.462260 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:18.462260 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:18.462260 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:41:18.462260 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:18.462260 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:41:18.462260 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:41:18.462260 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:18.462260 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:41:18.462260 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:18.462260 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:41:18.462260 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:41:18.462677 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:18.462677 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:41:18.462677 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:18.462677 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:41:18.462937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:18.462937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:41:18.462937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:41:18.462937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:41:18.462937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:41:18.462937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:41:18.462937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:18.462937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:41:18.462937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:41:18.462937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:18.462937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:41:18.462937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:41:18.462937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:18.462937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:18.462937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:41:18.462937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:18.462937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:41:18.463449 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:41:18.463449 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:41:18.463449 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:41:18.463449 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:18.463449 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:18.463449 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:18.463449 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:41:18.463449 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:18.463449 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:18.463449 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:18.463449 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:41:18.463449 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:18.463449 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:41:18.463449 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:41:18.463449 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:18.463449 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:41:18.463449 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:18.463449 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:41:18.463449 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:41:18.463979 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:18.463979 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:41:18.463979 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:18.463979 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:41:18.463979 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:41:18.463979 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:41:18.463979 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:41:18.463979 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:41:18.464272 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:41:18.464686 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:41:18.464888 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:18.465662 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:41:18.465662 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:18.465662 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:41:18.465662 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:41:18.465662 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:18.465662 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:41:18.465662 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:18.465662 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:41:18.466520 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:18.466520 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:41:18.466520 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:18.466520 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:41:18.466520 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:41:18.466520 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:41:18.466520 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:41:18.466520 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:18.466520 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:18.466520 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:18.466520 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:41:18.466520 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:18.466848 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:18.466848 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:18.466848 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:41:18.466848 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:18.466848 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:41:18.466848 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:41:18.466848 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:18.466848 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:41:18.466848 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:18.466848 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:41:18.466848 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:41:18.466848 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:18.466848 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:41:18.467192 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:18.467192 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:41:18.467510 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:18.467510 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:41:18.467510 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:41:18.467510 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:41:18.467510 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:41:18.467510 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:41:18.467510 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:18.467510 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:41:18.467510 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:41:18.467510 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:18.467510 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:41:18.467510 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:41:18.467510 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:18.467510 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:18.467510 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:41:18.467510 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:18.467988 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:41:18.467988 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:41:18.467988 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:18.467988 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:18.467988 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:18.467988 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:41:18.467988 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:18.467988 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:18.467988 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:18.467988 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:41:18.467988 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:18.467988 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:41:18.467988 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:41:18.467988 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:18.467988 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:41:18.467988 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:18.467988 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:41:18.467988 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:41:18.467988 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:18.468536 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:41:18.468536 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:18.468536 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:41:22.978813 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:41:22.979873 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:41:22.979873 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:41:22.980821 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:41:23.017033 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:41:23.017493 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:23.018732 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:41:23.018732 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:23.018732 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:41:23.018732 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:41:23.018732 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:23.018732 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:41:23.018732 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:23.018732 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:41:23.020779 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.039 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:23.020779 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:41:23.020779 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:23.020779 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:41:23.020779 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:41:23.020779 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:41:23.020779 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:41:23.020779 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:23.020779 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:23.020779 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:23.020779 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:41:23.020779 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:23.021511 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:23.021511 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:23.021511 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:41:23.021511 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:23.021511 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:41:23.021511 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:41:23.021511 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:23.021511 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:41:23.021511 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:23.021511 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:41:23.021511 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:41:23.021511 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:23.021511 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:41:23.022075 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:23.022075 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:41:23.022587 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:23.022587 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:41:23.022587 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:41:23.022587 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:41:23.022587 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:41:23.022587 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:41:23.022587 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:23.022587 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:41:23.022587 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:41:23.022587 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:23.022587 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:41:23.022587 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:41:23.022587 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:23.022587 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:23.022587 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:41:23.022587 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:23.023381 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:41:23.023381 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:41:23.023381 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:23.023381 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:23.023381 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:23.023381 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:41:23.023381 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:23.023381 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:23.023381 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:23.023381 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:41:23.023381 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:23.023381 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:41:23.023381 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:41:23.023381 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:23.023381 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:41:23.023381 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:23.023381 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:41:23.023381 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:41:23.023381 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:23.024223 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:41:23.024223 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:23.024223 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:41:23.442828 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:41:23.443610 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:41:23.443845 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:41:23.445121 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:41:23.445558 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:23.446470 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:41:23.446470 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:23.446470 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:41:23.446470 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:41:23.446470 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:23.446470 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:41:23.446470 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:23.446470 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:41:23.448231 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:23.448231 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:41:23.448231 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:23.448231 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:41:23.448231 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:41:23.448231 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:41:23.448231 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:41:23.448231 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:23.448231 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:23.448231 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:23.448231 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:41:23.448231 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:23.449189 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:23.449189 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:23.449189 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:41:23.449189 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:23.449189 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:41:23.449189 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:41:23.449189 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:23.449189 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:41:23.449189 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:23.449189 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:41:23.449189 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:41:23.449189 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:23.449189 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:41:23.450176 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:23.450176 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:41:23.450176 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:23.450176 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:41:23.450176 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:41:23.450176 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:41:23.450176 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:41:23.450176 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:41:23.450176 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:23.450176 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:41:23.450176 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:41:23.450176 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:23.450176 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:41:23.450176 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:41:23.450176 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:23.450976 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:23.450976 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:41:23.450976 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:23.450976 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:41:23.450976 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:41:23.450976 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:23.450976 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:23.450976 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:23.450976 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:41:23.450976 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:23.450976 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:23.450976 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:23.450976 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:41:23.450976 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:23.450976 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:41:23.450976 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:41:23.450976 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:23.450976 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:41:23.450976 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:23.451920 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:41:23.451920 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:41:23.451920 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:23.451920 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:41:23.451920 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:23.451920 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:41:23.451920 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:41:23.451920 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:41:23.451920 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:41:23.451920 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:41:23.452451 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:41:23.452451 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:23.452834 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:41:23.452834 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:23.452834 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:41:23.452834 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:41:23.452834 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:23.452834 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:41:23.452834 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:23.452834 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:41:23.453665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:23.453665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:41:23.453665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:41:23.453665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:41:23.453665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:23.453665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:41:23.453665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:23.453665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:41:23.453665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:23.453665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:41:23.453665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:23.453665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:41:23.453665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:23.453665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:41:23.453665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:41:23.454204 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:23.454204 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:41:23.454204 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:23.454204 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:41:23.454204 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:41:23.454204 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:23.454204 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:41:23.454204 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:23.454204 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:41:23.454525 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:23.454525 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:41:23.454525 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:23.454525 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:41:23.454525 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:41:23.454525 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:41:23.454525 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:41:23.454525 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:41:23.454525 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:41:23.454525 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:41:23.454525 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:23.454525 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:23.454896 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:23.454896 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:41:23.454896 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:23.454896 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:23.454896 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:23.454896 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:41:23.454896 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:23.454896 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:41:23.454896 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:41:23.454896 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:23.454896 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:41:23.454896 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:23.454896 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:41:23.454896 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:41:23.455306 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:23.455306 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:41:23.455306 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:23.455306 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:41:23.455702 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:23.455702 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:41:23.455702 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:41:23.455702 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:41:23.455702 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:41:23.455702 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:41:23.455702 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:23.455702 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:41:23.455702 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:41:23.455702 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:23.455702 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:41:23.455702 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:41:23.455702 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:23.455702 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:23.455702 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:41:23.455702 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:23.456130 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:41:23.456130 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:41:23.456130 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:41:23.456130 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:41:23.456130 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:41:23.456130 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:23.456130 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:23.456130 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:23.456130 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:41:23.456130 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:23.456130 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:23.456130 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:23.456130 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:41:23.456130 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:23.456130 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:41:23.456130 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:41:23.456130 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:23.456130 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:41:23.456647 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:23.456647 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:41:23.456647 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:41:23.456647 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:23.456647 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:41:23.456647 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:23.456647 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:41:23.458420 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:41:23.458739 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:41:23.458867 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:41:23.459024 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:41:23.459967 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:41:23.459967 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:23.460396 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:41:23.460396 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:23.460396 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:41:23.460396 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:41:23.460396 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:23.460396 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:41:23.460396 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:23.460396 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:41:23.461210 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:23.461210 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:41:23.461210 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:41:23.461210 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:41:23.461210 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:23.461210 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:41:23.461210 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:23.461210 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:41:23.461210 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:23.461210 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:41:23.461210 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:23.461210 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:41:23.461210 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:23.461210 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:41:23.461210 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:41:23.461956 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:23.461956 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:41:23.461956 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:23.461956 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:41:23.461956 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:41:23.461956 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:23.461956 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:41:23.461956 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:23.461956 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:41:23.462140 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:23.462140 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:41:23.462140 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:23.462140 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:41:23.462140 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:41:23.462140 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:41:23.462140 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:41:23.462140 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:41:23.462140 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:41:23.462140 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:23.462140 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:23.462369 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:23.462369 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:41:23.462369 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:23.462369 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:23.462369 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:23.462369 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:41:23.462369 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:23.462369 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:41:23.462369 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:41:23.462369 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:23.462369 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:41:23.462369 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:23.462369 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:41:23.462369 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:41:23.462649 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:23.462649 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:41:23.462649 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:23.462649 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:41:23.463310 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:23.463310 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:41:23.463310 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:41:23.463310 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:41:23.463310 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:41:23.463310 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:41:23.463310 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:23.463310 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:41:23.463310 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:41:23.463310 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:23.463310 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:41:23.463310 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:41:23.463310 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:23.463310 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:23.463310 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:41:23.463310 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:23.463310 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:41:23.463693 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:41:23.463693 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:41:23.463693 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:41:23.463693 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:23.463693 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:23.463693 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:23.463693 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:41:23.463693 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:23.463693 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:23.463693 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:23.463693 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:41:23.463693 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:23.463693 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:41:23.463693 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:41:23.463693 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:23.463693 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:41:23.463693 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:23.463693 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:41:23.463693 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:41:23.464083 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:23.464083 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:41:23.464083 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:23.464083 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:41:23.464083 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:41:23.464226 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:41:23.464226 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:41:23.464421 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:41:23.464787 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:41:23.465026 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:41:23.465168 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:23.465749 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:41:23.465749 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:23.465749 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:41:23.465749 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:41:23.465749 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:23.465749 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:41:23.465749 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:23.465749 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:41:23.466659 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:23.466659 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:41:23.466659 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:23.466659 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:41:23.466659 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:41:23.466659 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:41:23.466659 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:41:23.466659 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:23.466659 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:23.466659 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:23.466659 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:41:23.466659 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:23.467187 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:23.467187 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:23.467187 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:41:23.467187 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:23.467187 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:41:23.467187 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:41:23.467187 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:23.467187 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:41:23.467187 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:23.467187 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:41:23.467187 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:41:23.467187 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:23.467187 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:41:23.467741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:23.467741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:41:23.467836 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:23.467836 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:41:23.467836 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:41:23.467836 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:41:23.467836 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:41:23.467836 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:41:23.467836 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:23.467836 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:41:23.467836 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:41:23.467836 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:23.467836 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:41:23.467836 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:41:23.467836 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:23.467836 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:23.467836 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:41:23.467836 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:23.468458 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:41:23.468458 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:41:23.468458 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:23.468458 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:23.468458 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:23.468458 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:41:23.468458 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:23.468458 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:23.468458 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:23.468458 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:41:23.468458 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:23.468458 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:41:23.468458 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:41:23.468458 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:23.468458 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:41:23.468458 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:23.468458 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:41:23.468458 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:41:23.468458 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:23.469193 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:41:23.469193 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:23.469193 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:41:27.979351 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:41:27.980143 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:41:27.980252 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:41:27.981442 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:41:28.019115 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:41:28.019478 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:28.020315 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:41:28.020315 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:28.020315 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:41:28.020315 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:41:28.020315 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:28.020315 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:41:28.020315 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:28.020315 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:41:28.021369 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.040 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:28.021369 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:41:28.021369 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:28.021369 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:41:28.021369 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:41:28.021369 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:41:28.021369 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:41:28.021369 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:28.021369 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:28.021369 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:28.021369 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:41:28.021369 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:28.022220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:28.022220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:28.022220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:41:28.022220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:28.022220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:41:28.022220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:41:28.022220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:28.022220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:41:28.022220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:28.022220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:41:28.022220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:41:28.022220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:28.022220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:41:28.023116 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:28.023116 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:41:28.023116 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:28.023116 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:41:28.023116 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:41:28.023116 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:41:28.023116 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:41:28.023116 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:41:28.023116 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:28.023116 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:41:28.023116 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:41:28.023116 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:28.023116 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:41:28.023116 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:41:28.023116 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:28.024167 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:28.024167 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:41:28.024167 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:28.024167 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:41:28.024167 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:41:28.024167 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:28.024167 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:28.024167 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:28.024167 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:41:28.024167 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:28.024167 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:28.024167 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:28.024167 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:41:28.024167 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:28.024167 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:41:28.024167 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:41:28.024167 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:28.024167 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:41:28.024167 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:28.026664 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:41:28.026664 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:41:28.026664 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:28.026664 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:41:28.026664 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:28.026664 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:41:28.443723 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:41:28.444439 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:41:28.444566 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:41:28.445832 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:41:28.446235 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:28.447182 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:41:28.447182 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:28.447182 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:41:28.447182 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:41:28.447182 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:28.447182 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:41:28.447182 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:28.447182 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:41:28.448667 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:28.448667 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:41:28.448667 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:28.448667 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:41:28.448667 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:41:28.448667 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:41:28.448667 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:41:28.448667 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:28.448667 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:28.448667 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:28.448667 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:41:28.448667 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:28.449368 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:28.449368 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:28.449368 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:41:28.449368 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:28.449368 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:41:28.449368 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:41:28.449368 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:28.449368 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:41:28.449368 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:28.449368 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:41:28.449368 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:41:28.449368 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:28.449368 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:41:28.450130 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:28.450130 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:41:28.450429 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:28.450429 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:41:28.450429 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:41:28.450429 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:41:28.450429 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:41:28.450429 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:41:28.450429 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:28.450429 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:41:28.450429 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:41:28.450429 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:28.450429 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:41:28.450429 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:41:28.450429 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:28.450429 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:28.450429 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:41:28.450429 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:28.451147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:41:28.451147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:41:28.451147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:28.451147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:28.451147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:28.451147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:41:28.451147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:28.451147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:28.451147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:28.451147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:41:28.451147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:28.451147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:41:28.451147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:41:28.451147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:28.451147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:41:28.451147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:28.451147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:41:28.451147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:41:28.451147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:28.451959 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:41:28.451959 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:28.451959 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:41:28.451959 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:41:28.451959 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:41:28.451959 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:41:28.452350 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:41:28.452652 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:41:28.453041 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:28.453938 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:41:28.453938 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:28.453938 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:41:28.453938 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:41:28.453938 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:28.453938 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:41:28.453938 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:28.453938 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:41:28.455186 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:28.455186 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:41:28.455186 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:41:28.455186 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:41:28.455186 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:28.455186 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:41:28.455186 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:28.455186 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:41:28.455186 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:28.455186 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:41:28.455186 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:28.455186 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:41:28.455186 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:28.455186 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:41:28.455186 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:41:28.455773 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:28.455773 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:41:28.455773 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:28.455773 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:41:28.455773 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:41:28.455773 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:28.455773 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:41:28.455773 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:28.455773 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:41:28.456564 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:28.456564 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:41:28.456564 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:28.456564 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:41:28.456564 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:41:28.456564 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:41:28.456564 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:41:28.456564 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:41:28.456564 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:41:28.456564 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:41:28.456564 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:28.456564 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:28.457028 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:28.457028 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:41:28.457028 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:28.457028 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:28.457028 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:28.457028 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:41:28.457028 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:28.457028 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:41:28.457028 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:41:28.457028 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:28.457028 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:41:28.457028 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:28.457028 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:41:28.457028 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:41:28.457489 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:28.457489 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:41:28.457489 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:28.457489 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:41:28.458194 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:28.458194 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:41:28.458194 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:41:28.458194 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:41:28.458194 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:41:28.458194 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:41:28.458194 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:28.458194 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:41:28.458194 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:41:28.458194 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:28.458194 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:41:28.458194 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:41:28.458194 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:28.458194 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:28.458194 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:41:28.458194 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:28.458697 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:41:28.458697 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:41:28.458697 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:41:28.458697 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:41:28.458697 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:41:28.458697 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:28.458697 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:28.458697 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:28.458697 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:41:28.458697 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:28.458697 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:28.458697 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:28.458697 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:41:28.458697 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:28.458697 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:41:28.458697 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:41:28.458697 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:28.458697 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:41:28.459220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:28.459220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:41:28.459220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:41:28.459220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:28.459220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:41:28.459220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:28.459220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:41:28.459220 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:41:28.459478 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:41:28.459478 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:41:28.459624 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:41:28.460033 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:41:28.460282 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:28.460956 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:41:28.460956 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:28.460956 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:41:28.460956 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:41:28.460956 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:28.460956 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:41:28.460956 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:28.460956 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:41:28.461701 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:28.461701 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:41:28.461701 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:41:28.461701 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:41:28.461701 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:28.461701 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:41:28.461701 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:28.461701 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:41:28.461701 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:28.461701 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:41:28.461701 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:28.461701 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:41:28.461701 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:28.461701 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:41:28.461701 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:41:28.462357 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:28.462357 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:41:28.462357 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:28.462357 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:41:28.462357 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:41:28.462357 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:28.462357 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:41:28.462357 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:28.462357 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:41:28.462712 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:28.462712 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:41:28.462712 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:28.462712 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:41:28.462712 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:41:28.462712 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:41:28.462712 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:41:28.462712 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:41:28.462712 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:41:28.462712 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:28.462712 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:28.463136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:28.463136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:41:28.463136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:28.463136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:28.463136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:28.463136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:41:28.463136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:28.463136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:41:28.463136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:41:28.463136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:28.463136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:41:28.463136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:28.463136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:41:28.463136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:41:28.463857 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:28.463857 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:41:28.463857 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:28.463857 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:41:28.463857 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:28.463857 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:41:28.463857 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:41:28.463857 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:41:28.463857 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:41:28.463857 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:41:28.463857 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:28.463857 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:41:28.463857 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:41:28.463857 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:28.464884 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:41:28.464884 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:41:28.464884 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:28.464884 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:28.464884 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:41:28.464884 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:28.464884 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:41:28.464884 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:41:28.464884 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:41:28.464884 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:41:28.464884 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:28.464884 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:28.464884 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:28.464884 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:41:28.464884 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:28.464884 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:28.464884 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:28.464884 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:41:28.464884 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:28.465895 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:41:28.465895 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:41:28.465895 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:28.465895 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:41:28.465895 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:28.465895 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:41:28.465895 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:41:28.465895 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:28.465895 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:41:28.465895 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:28.465895 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:41:28.465895 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:41:28.465895 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:41:28.466747 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:41:28.466747 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:41:28.466747 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:41:28.466747 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:41:28.466747 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:28.466747 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:41:28.466747 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:28.466747 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:41:28.466747 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:41:28.467304 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:28.467304 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:41:28.467304 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:28.467304 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:41:28.467734 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:28.467734 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:41:28.467734 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:28.467734 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:41:28.467734 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:41:28.467734 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:41:28.467734 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:41:28.467734 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:28.467734 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:28.467734 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:28.467734 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:41:28.467734 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:28.468621 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:28.468621 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:28.468621 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:41:28.468621 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:28.468621 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:41:28.468621 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:41:28.468621 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:28.468621 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:41:28.468621 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:28.468621 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:41:28.468621 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:41:28.468621 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:28.468621 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:41:28.469634 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:28.469634 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:41:28.469634 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:28.469634 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:41:28.469634 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:41:28.469634 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:41:28.469634 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:41:28.469634 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:41:28.469634 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:28.469634 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:41:28.469634 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:41:28.469634 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:28.469634 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:41:28.469634 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:41:28.469634 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:28.470522 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:28.470522 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:41:28.470522 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:28.470522 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:41:28.470522 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:41:28.470522 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:28.470522 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:28.470522 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:28.470522 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:41:28.470522 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:28.470522 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:28.470522 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:28.470522 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:41:28.470522 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:28.470522 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:41:28.470522 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:41:28.470522 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:28.470522 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:41:28.470522 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:28.471359 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:41:28.471359 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:41:28.471359 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:28.471359 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:41:28.471359 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:28.471359 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:41:32.980386 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:41:32.983962 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:41:32.984070 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:41:32.984892 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:41:33.012966 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:41:33.013302 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:33.014111 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:41:33.014111 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:33.014111 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:41:33.014111 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:41:33.014111 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:33.014111 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:41:33.014111 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:33.014111 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:41:33.015081 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.030 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:33.015081 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:41:33.015081 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:33.015081 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:41:33.015081 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:41:33.015081 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:41:33.015081 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:41:33.015081 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:33.015081 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:33.015081 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:33.015081 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:41:33.015081 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:33.015559 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:33.015559 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:33.015559 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:41:33.015559 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:33.015559 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:41:33.015559 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:41:33.015559 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:33.015559 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:41:33.015559 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:33.015559 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:41:33.015559 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:41:33.015559 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:33.015559 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:41:33.016137 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:33.016137 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:41:33.016137 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:33.016137 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:41:33.016137 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:41:33.016137 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:41:33.016137 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:41:33.016137 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:41:33.016137 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:33.016137 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:41:33.016137 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:41:33.016137 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:33.016137 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:41:33.016137 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:41:33.016137 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:33.016819 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:33.016819 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:41:33.016819 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:33.016819 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:41:33.016819 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:41:33.016819 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:33.016819 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:33.016819 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:33.016819 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:41:33.016819 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:33.016819 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:33.016819 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:33.016819 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:41:33.016819 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:33.016819 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:41:33.016819 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:41:33.016819 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:33.016819 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:41:33.016819 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:33.017858 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:41:33.017858 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:41:33.017858 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:33.017858 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:41:33.017858 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:33.017858 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:41:33.445411 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:41:33.446026 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:41:33.446156 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:41:33.447722 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:41:33.447722 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:33.447909 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:41:33.447909 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:33.447909 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:41:33.447909 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:41:33.447909 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:33.447909 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:41:33.447909 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:33.447909 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:41:33.449161 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:33.449161 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:41:33.449161 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:33.449161 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:41:33.449161 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:41:33.449161 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:41:33.449161 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:41:33.449161 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:33.449161 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:33.449161 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:33.449161 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:41:33.449161 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:33.449578 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:33.449578 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:33.449578 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:41:33.449578 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:33.449578 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:41:33.449578 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:41:33.449578 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:33.449578 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:41:33.449578 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:33.449578 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:41:33.449578 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:41:33.449578 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:33.449578 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:41:33.449922 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:33.449922 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:41:33.450252 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:33.450252 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:41:33.450252 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:41:33.450252 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:41:33.450252 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:41:33.450252 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:41:33.450252 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:33.450252 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:41:33.450252 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:41:33.450252 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:33.450252 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:41:33.450252 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:41:33.450252 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:33.450252 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:33.450252 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:41:33.450252 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:33.450829 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:41:33.450829 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:41:33.450829 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:33.450829 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:33.450829 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:33.450829 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:41:33.450829 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:33.450829 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:33.450829 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:33.450829 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:41:33.450829 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:33.450829 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:41:33.450829 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:41:33.450829 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:33.450829 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:41:33.450829 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:33.450829 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:41:33.450829 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:41:33.450829 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:33.451688 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:41:33.451688 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:33.451688 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:41:33.452820 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:41:33.453151 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:41:33.453269 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:41:33.453401 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:41:33.453860 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:41:33.454285 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:33.455187 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:41:33.455187 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:33.455187 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:41:33.455187 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:41:33.455187 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:33.455187 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:41:33.455187 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:33.455187 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:41:33.455975 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:33.455975 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:41:33.455975 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:41:33.455975 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:41:33.455975 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:33.455975 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:41:33.455975 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:33.455975 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:41:33.455975 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:33.455975 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:41:33.455975 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:33.455975 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:41:33.455975 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:33.455975 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:41:33.455975 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:41:33.456446 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:33.456446 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:41:33.456446 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:33.456446 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:41:33.456446 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:41:33.456446 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:33.456446 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:41:33.456446 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:33.456446 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:41:33.456851 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:33.456851 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:41:33.456851 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:33.456851 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:41:33.456851 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:41:33.456851 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:41:33.456851 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:41:33.456851 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:41:33.456851 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:41:33.456851 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:41:33.456851 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:33.456851 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:33.457196 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:33.457196 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:41:33.457196 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:33.457196 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:33.457196 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:33.457196 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:41:33.457196 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:33.457196 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:41:33.457196 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:41:33.457196 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:33.457196 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:41:33.457196 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:33.457196 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:41:33.457196 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:41:33.457627 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:33.457627 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:41:33.457627 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:33.457627 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:41:33.457914 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:33.457914 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:41:33.457914 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:41:33.457914 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:41:33.457914 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:41:33.457914 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:41:33.457914 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:33.457914 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:41:33.457914 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:41:33.457914 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:33.457914 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:41:33.457914 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:41:33.457914 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:33.457914 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:33.457914 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:41:33.457914 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:33.458394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:41:33.458394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:41:33.458394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:41:33.458394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:41:33.458394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:41:33.458394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:33.458394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:33.458394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:33.458394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:41:33.458394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:33.458394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:33.458394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:33.458394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:41:33.458394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:33.458394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:41:33.458394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:41:33.458394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:33.458394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:41:33.458930 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:33.458930 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:41:33.458930 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:41:33.458930 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:33.458930 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:41:33.458930 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:33.458930 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:41:33.460154 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:41:33.460506 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:41:33.460616 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:41:33.460748 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:41:33.461715 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:41:33.461715 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:33.461878 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:41:33.461878 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:33.461878 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:41:33.461878 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:41:33.461878 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:33.461878 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:41:33.461878 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:33.461878 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:41:33.462649 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:33.462649 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:41:33.462649 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:41:33.462649 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:41:33.462649 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:33.462649 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:41:33.462649 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:33.462649 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:41:33.462649 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:33.462649 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:41:33.462649 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:33.462649 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:41:33.462649 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:33.462649 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:41:33.462649 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:41:33.463121 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:33.463121 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:41:33.463121 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:33.463121 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:41:33.463121 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:41:33.463121 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:33.463121 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:41:33.463121 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:33.463121 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:41:33.463603 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:33.463603 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:41:33.463603 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:33.463603 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:41:33.463603 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:41:33.463603 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:41:33.463603 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:41:33.463603 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:41:33.463603 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:41:33.463603 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:33.463603 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:33.463957 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:33.463957 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:41:33.463957 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:33.463957 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:33.463957 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:33.463957 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:41:33.463957 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:33.463957 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:41:33.463957 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:41:33.463957 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:33.463957 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:41:33.463957 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:33.463957 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:41:33.463957 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:41:33.464388 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:33.464388 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:41:33.464388 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:33.464388 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:41:33.464770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:33.464770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:41:33.464770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:41:33.464770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:41:33.464770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:41:33.464770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:41:33.464770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:33.464770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:41:33.464770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:41:33.464770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:33.464770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:41:33.464770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:41:33.464770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:33.464770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:33.464770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:41:33.464770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:33.464770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:41:33.465253 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:41:33.465253 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:41:33.465253 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:41:33.465253 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:33.465253 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:33.465253 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:33.465253 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:41:33.465253 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:33.465253 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:33.465253 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:33.465253 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:41:33.465253 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:33.465253 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:41:33.465253 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:41:33.465253 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:33.465253 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:41:33.465253 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:33.465253 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:41:33.465253 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:41:33.465864 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:33.465864 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:41:33.465864 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:33.465864 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:41:33.465864 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:41:33.465864 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:41:33.465864 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:41:33.465864 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:41:33.466834 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:41:33.466834 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:41:33.467074 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:33.467744 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:41:33.467744 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:33.467744 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:41:33.467744 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:41:33.467744 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:33.467744 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:41:33.467744 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:33.467744 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:41:33.469220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:33.469220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:41:33.469220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:33.469220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:41:33.469220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:41:33.469220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:41:33.469220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:41:33.469220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:33.469220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:33.469220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:33.469220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:41:33.469220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:33.469767 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:33.469767 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:33.469767 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:41:33.469767 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:33.469767 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:41:33.469767 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:41:33.469767 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:33.469767 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:41:33.469767 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:33.469767 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:41:33.469767 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:41:33.469767 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:33.469767 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:41:33.470217 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:33.470217 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:41:33.470313 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:33.470313 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:41:33.470313 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:41:33.470313 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:41:33.470313 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:41:33.470313 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:41:33.470313 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:33.470313 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:41:33.470313 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:41:33.470313 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:33.470313 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:41:33.470313 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:41:33.470313 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:33.470313 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:33.470313 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:41:33.470313 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:33.470854 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:41:33.470854 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:41:33.470854 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:33.470854 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:33.470854 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:33.470854 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:41:33.470854 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:33.470854 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:33.470854 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:33.470854 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:41:33.470854 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:33.470854 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:41:33.470854 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:41:33.470854 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:33.470854 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:41:33.470854 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:33.470854 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:41:33.470854 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:41:33.470854 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:33.471469 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:41:33.471469 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:33.471469 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:41:37.984015 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:41:37.984924 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:41:37.984924 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:41:37.985909 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:41:38.013095 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:41:38.013608 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:38.014631 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:41:38.014631 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:38.014631 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:41:38.014631 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:41:38.014631 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:38.014631 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:41:38.014631 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:38.014631 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:41:38.015480 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.030 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:38.015480 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:41:38.015480 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:38.015480 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:41:38.015480 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:41:38.015480 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:41:38.015480 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:41:38.015480 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:38.015480 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:38.015480 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:38.015480 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:41:38.015480 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:38.016104 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:38.016104 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:38.016104 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:41:38.016104 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:38.016104 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:41:38.016104 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:41:38.016104 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:38.016104 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:41:38.016104 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:38.016104 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:41:38.016104 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:41:38.016104 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:38.016104 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:41:38.016770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:38.016770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:41:38.016770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:38.016770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:41:38.016770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:41:38.016770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:41:38.016770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:41:38.016770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:41:38.016770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:38.016770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:41:38.016770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:41:38.016770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:38.016770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:41:38.016770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:41:38.016770 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:38.017522 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:38.017522 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:41:38.017522 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:38.017522 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:41:38.017522 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:41:38.017522 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:38.017522 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:38.017522 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:38.017522 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:41:38.017522 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:38.017522 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:38.017522 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:38.017522 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:41:38.017522 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:38.017522 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:41:38.017522 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:41:38.017522 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:38.017522 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:41:38.017522 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:38.018858 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:41:38.018858 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:41:38.018858 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:38.018858 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:41:38.018858 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:38.018858 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:41:38.446269 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:41:38.447613 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:41:38.447919 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:41:38.449196 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:41:38.449556 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:38.450548 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:41:38.450548 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:38.450548 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:41:38.450548 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:41:38.450548 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:38.450548 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:41:38.450548 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:38.450548 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:41:38.452901 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:38.452901 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:41:38.452901 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:38.452901 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:41:38.452901 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:41:38.452901 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:41:38.452901 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:41:38.452901 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:38.452901 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:38.452901 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:38.452901 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:41:38.452901 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:38.453379 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:38.453379 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:38.453379 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:41:38.453379 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:38.453379 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:41:38.453379 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:41:38.453379 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:38.453379 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:41:38.453379 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:38.453379 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:41:38.453379 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:41:38.453379 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:38.453379 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:41:38.453838 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:38.453838 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:41:38.454847 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:38.454847 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:41:38.454847 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:41:38.454847 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:41:38.454847 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:41:38.454847 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:41:38.454847 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:38.454847 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:41:38.454847 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:41:38.454847 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:38.454847 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:41:38.454847 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:41:38.454847 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:38.454847 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:38.454847 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:41:38.454847 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:38.455477 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:41:38.455477 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:41:38.455477 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:38.455477 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:38.455477 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:38.455477 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:41:38.455477 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:38.455477 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:38.455477 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:38.455477 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:41:38.455477 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:38.455477 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:41:38.455477 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:41:38.455477 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:38.455477 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:41:38.455477 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:38.455477 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:41:38.455477 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:41:38.455477 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:38.456078 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:41:38.456078 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:38.456078 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:41:38.456078 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:41:38.456078 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:41:38.456078 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:41:38.456078 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:41:38.456335 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:41:38.457342 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:38.457342 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:41:38.457342 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:38.457342 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:41:38.457342 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:41:38.457342 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:38.457342 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:41:38.457342 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:38.457342 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:41:38.458122 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:38.458122 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:41:38.458122 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:41:38.458122 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:41:38.458122 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:38.458122 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:41:38.458122 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:38.458122 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:41:38.458122 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:38.458122 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:41:38.458122 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:38.458122 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:41:38.458122 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:38.458122 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:41:38.458122 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:41:38.458598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:38.458598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:41:38.458598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:38.458598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:41:38.458598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:41:38.458598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:38.458598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:41:38.458598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:38.458598 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:41:38.458850 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:38.458850 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:41:38.458850 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:38.458850 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:41:38.458850 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:41:38.458850 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:41:38.458850 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:41:38.458850 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:41:38.458850 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:41:38.458850 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:41:38.458850 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:38.458850 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:38.459199 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:38.459199 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:41:38.459199 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:38.459199 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:38.459199 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:38.459199 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:41:38.459199 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:38.459199 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:41:38.459199 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:41:38.459199 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:38.459199 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:41:38.459199 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:38.459199 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:41:38.459199 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:41:38.459584 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:38.459584 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:41:38.459584 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:38.459584 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:41:38.459584 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:38.459584 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:41:38.459584 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:41:38.459584 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:41:38.459584 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:41:38.459584 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:41:38.459584 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:38.459584 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:41:38.459584 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:41:38.459584 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:38.460002 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:41:38.460002 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:41:38.460002 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:38.460002 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:38.460002 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:41:38.460002 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:38.460002 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:41:38.460002 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:41:38.460002 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:41:38.460002 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:41:38.460002 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:41:38.460002 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:38.460002 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:38.460002 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:38.460002 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:41:38.460002 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:38.460002 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:38.460002 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:38.460002 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:41:38.460609 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:38.460609 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:41:38.460609 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:41:38.460609 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:38.460609 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:41:38.460609 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:38.460609 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:41:38.460609 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:41:38.460609 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:38.460609 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:41:38.460609 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:38.460609 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:41:38.460964 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:41:38.461016 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:41:38.461058 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:41:38.461172 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:41:38.461643 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:41:38.461990 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:38.462676 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:41:38.462676 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:38.462676 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:41:38.462676 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:41:38.462676 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:38.462676 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:41:38.462676 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:38.462676 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:41:38.463393 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:38.463393 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:41:38.463393 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:41:38.463393 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:41:38.463393 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:38.463393 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:41:38.463393 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:38.463393 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:41:38.463393 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:38.463393 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:41:38.463393 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:38.463393 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:41:38.463393 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:38.463393 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:41:38.463393 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:41:38.463812 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:38.463812 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:41:38.463812 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:38.463812 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:41:38.463812 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:41:38.463812 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:38.463812 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:41:38.463812 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:38.463812 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:41:38.464505 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:38.464505 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:41:38.464505 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:38.464505 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:41:38.464505 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:41:38.464505 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:41:38.464505 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:41:38.464505 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:41:38.464505 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:41:38.464505 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:38.464505 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:38.464897 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:38.464897 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:41:38.464897 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:38.464897 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:38.464897 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:38.464897 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:41:38.464897 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:38.464897 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:41:38.464897 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:41:38.464897 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:38.464897 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:41:38.464897 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:38.464897 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:41:38.464897 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:41:38.465349 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:38.465349 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:41:38.465349 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:38.465349 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:41:38.465689 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:38.465689 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:41:38.465689 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:41:38.465689 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:41:38.465689 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:41:38.465689 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:41:38.465689 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:38.465689 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:41:38.465689 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:41:38.465689 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:38.465689 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:41:38.465689 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:41:38.465689 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:38.465689 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:38.465689 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:41:38.465689 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:38.465689 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:41:38.466205 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:41:38.466205 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:41:38.466205 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:41:38.466205 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:38.466205 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:38.466205 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:38.466205 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:41:38.466205 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:38.466205 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:38.466205 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:38.466205 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:41:38.466205 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:38.466205 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:41:38.466205 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:41:38.466205 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:38.466205 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:41:38.466205 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:38.466205 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:41:38.466205 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:41:38.466787 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:38.466787 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:41:38.466787 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:38.466787 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:41:38.466787 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:41:38.466958 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:41:38.467125 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:41:38.467351 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:41:38.467822 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:41:38.468162 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:41:38.468359 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:38.469545 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:41:38.469545 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:38.469545 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:41:38.469545 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:41:38.469545 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:38.469545 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:41:38.469545 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:38.469545 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:41:38.470145 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:38.470145 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:41:38.470145 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:38.470145 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:41:38.470145 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:41:38.470145 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:41:38.470145 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:41:38.470145 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:38.470145 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:38.470145 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:38.470145 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:41:38.470145 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:38.470624 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:38.470624 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:38.470624 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:41:38.470624 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:38.470624 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:41:38.470624 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:41:38.470624 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:38.470624 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:41:38.470624 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:38.470624 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:41:38.470624 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:41:38.470624 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:38.470624 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:41:38.470987 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:38.470987 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:41:38.471121 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:38.471121 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:41:38.471121 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:41:38.471121 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:41:38.471121 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:41:38.471121 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:41:38.471121 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:38.471121 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:41:38.471121 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:41:38.471121 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:38.471121 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:41:38.471121 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:41:38.471121 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:38.471121 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:38.471121 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:41:38.471121 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:38.471578 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:41:38.471578 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:41:38.471578 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:38.471578 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:38.471578 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:38.471578 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:41:38.471578 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:38.471578 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:38.471578 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:38.471578 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:41:38.471578 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:38.471578 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:41:38.471578 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:41:38.471578 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:38.471578 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:41:38.471578 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:38.471578 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:41:38.471578 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:41:38.471578 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:38.472150 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:41:38.472150 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:38.472150 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:41:42.984634 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:41:42.985421 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:41:42.985421 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:41:42.985874 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:41:43.014829 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:41:43.015283 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:43.016264 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:41:43.016264 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:43.016264 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:41:43.016264 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:41:43.016264 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:43.016264 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:41:43.016264 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:43.016264 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:41:43.017219 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.031 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:43.017219 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:41:43.017219 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:43.017219 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:41:43.017219 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:41:43.017219 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:41:43.017219 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:41:43.017219 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:43.017219 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:43.017219 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:43.017219 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:41:43.017219 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:43.017872 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:43.017872 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:43.017872 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:41:43.017872 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:43.017872 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:41:43.017872 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:41:43.017872 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:43.017872 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:41:43.017872 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:43.017872 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:41:43.017872 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:41:43.017872 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:43.017872 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:41:43.018542 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:43.018542 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:41:43.018542 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:43.018542 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:41:43.018542 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:41:43.018542 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:41:43.018542 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:41:43.018542 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:41:43.018542 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:43.018542 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:41:43.018542 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:41:43.018542 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:43.018542 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:41:43.018542 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:41:43.018542 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:43.019176 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:43.019176 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:41:43.019176 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:43.019176 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:41:43.019176 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:41:43.019176 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:43.019176 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:43.019176 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:43.019176 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:41:43.019176 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:43.019176 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:43.019176 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:43.019176 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:41:43.019176 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:43.019176 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:41:43.019176 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:41:43.019176 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:43.019176 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:41:43.019176 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:43.020021 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:41:43.020021 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:41:43.020021 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:43.020021 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:41:43.020021 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:43.020021 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:41:43.447253 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:41:43.447846 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:41:43.447967 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:41:43.448941 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:41:43.449447 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:43.450443 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:41:43.450443 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:43.450443 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:41:43.450443 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:41:43.450443 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:43.450443 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:41:43.450443 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:43.450443 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:41:43.451407 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:43.451407 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:41:43.451407 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:43.451407 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:41:43.451407 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:41:43.451407 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:41:43.451407 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:41:43.451407 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:43.451407 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:43.451407 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:43.451407 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:41:43.451407 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:43.451738 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:43.451738 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:43.451738 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:41:43.451738 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:43.451738 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:41:43.451738 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:41:43.451738 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:43.451738 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:41:43.451738 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:43.451738 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:41:43.451738 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:41:43.451738 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:43.451738 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:41:43.452099 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:43.452099 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:41:43.452436 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:43.452436 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:41:43.452436 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:41:43.452436 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:41:43.452436 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:41:43.452436 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:41:43.452436 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:43.452436 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:41:43.452436 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:41:43.452436 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:43.452436 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:41:43.452436 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:41:43.452436 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:43.452436 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:43.452436 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:41:43.452436 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:43.453073 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:41:43.453073 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:41:43.453073 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:43.453073 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:43.453073 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:43.453073 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:41:43.453073 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:43.453073 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:43.453073 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:43.453073 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:41:43.453073 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:43.453073 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:41:43.453073 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:41:43.453073 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:43.453073 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:41:43.453073 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:43.453073 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:41:43.453073 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:41:43.453073 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:43.453624 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:41:43.453624 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:43.453624 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:41:43.455591 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:41:43.455973 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:41:43.456237 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:41:43.456548 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:41:43.456949 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:41:43.457196 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:43.457801 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:41:43.457801 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:43.457801 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:41:43.457801 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:41:43.457801 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:43.457801 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:41:43.457801 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:43.457801 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:41:43.458635 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:43.458635 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:41:43.458635 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:41:43.458635 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:41:43.458635 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:43.458635 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:41:43.458635 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:43.458635 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:41:43.458635 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:43.458635 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:41:43.458635 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:43.458635 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:41:43.458635 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:43.458635 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:41:43.458635 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:41:43.459398 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:43.459398 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:41:43.459398 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:43.459398 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:41:43.459398 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:41:43.459398 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:43.459398 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:41:43.459398 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:43.459398 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:41:43.459398 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:43.459398 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:41:43.459398 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:43.459398 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:41:43.459398 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:41:43.460059 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:41:43.460059 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:41:43.460059 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:41:43.460059 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:41:43.460059 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:41:43.460059 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:43.460059 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:43.460059 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:43.460059 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:41:43.460059 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:43.460059 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:43.460059 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:43.460059 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:41:43.460665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:43.460665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:41:43.460665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:41:43.460665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:43.460665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:41:43.460665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:43.460665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:41:43.460665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:41:43.460665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:43.460665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:41:43.460665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:43.460665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:41:43.461217 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:43.461217 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:41:43.461217 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:41:43.461217 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:41:43.461217 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:41:43.461217 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:41:43.461217 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:43.461217 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:41:43.461217 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:41:43.461217 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:43.461217 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:41:43.461217 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:41:43.461217 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:43.461217 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:43.461217 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:41:43.461217 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:43.461915 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:41:43.461915 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:41:43.461915 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:41:43.461915 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:41:43.461915 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:41:43.461915 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:43.461915 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:43.461915 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:43.461915 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:41:43.461915 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:43.461915 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:43.461915 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:43.461915 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:41:43.461915 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:43.461915 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:41:43.461915 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:41:43.461915 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:43.461915 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:41:43.462750 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:43.462750 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:41:43.462750 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:41:43.462750 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:43.462750 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:41:43.462750 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:43.462750 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:41:43.462750 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:41:43.462750 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:41:43.462750 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:41:43.463022 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:41:43.463022 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:41:43.463022 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:43.463022 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:41:43.463022 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:43.463022 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:41:43.463022 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:41:43.463022 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:43.463022 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:41:43.463022 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:43.463022 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:41:43.463730 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:43.463730 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:41:43.463730 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:41:43.463730 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:41:43.463730 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:43.463730 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:41:43.463730 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:43.463730 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:41:43.463730 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:43.463730 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:41:43.463730 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:43.463730 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:41:43.463730 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:43.463730 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:41:43.463730 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:41:43.464153 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:43.464153 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:41:43.464153 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:43.464153 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:41:43.464153 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:41:43.464153 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:43.464153 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:41:43.464153 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:43.464153 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:41:43.464624 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:43.464624 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:41:43.464624 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:43.464624 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:41:43.464624 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:41:43.464624 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:41:43.464624 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:41:43.464624 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:41:43.464624 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:41:43.464624 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:43.464624 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:43.464937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:43.464937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:41:43.464937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:43.464937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:43.464937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:43.464937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:41:43.464937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:43.464937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:41:43.464937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:41:43.464937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:43.464937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:41:43.464937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:43.464937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:41:43.464937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:41:43.465337 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:43.465337 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:41:43.465337 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:43.465337 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:41:43.465656 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:43.465656 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:41:43.465656 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:41:43.465656 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:41:43.465656 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:41:43.465656 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:41:43.465656 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:43.465656 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:41:43.465656 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:41:43.465656 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:43.465656 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:41:43.465656 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:41:43.465656 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:43.465656 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:43.465656 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:41:43.465656 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:43.465656 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:41:43.466098 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:41:43.466098 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:41:43.466098 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:41:43.466098 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:43.466098 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:43.466098 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:43.466098 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:41:43.466098 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:43.466098 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:43.466098 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:43.466098 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:41:43.466098 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:43.466098 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:41:43.466098 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:41:43.466098 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:43.466098 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:41:43.466098 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:43.466098 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:41:43.466098 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:41:43.466675 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:43.466675 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:41:43.466675 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:43.466675 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:41:43.466675 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:41:43.466839 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:41:43.466977 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:41:43.467111 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:41:43.467641 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:41:43.468016 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:41:43.468212 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:43.469030 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:41:43.469030 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:43.469030 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:41:43.469030 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:41:43.469030 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:43.469030 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:41:43.469030 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:43.469030 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:41:43.469835 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:43.469835 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:41:43.469835 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:43.469835 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:41:43.469835 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:41:43.469835 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:41:43.469835 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:41:43.469835 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:43.469835 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:43.469835 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:43.469835 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:41:43.469835 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:43.470168 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:43.470168 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:43.470168 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:41:43.470168 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:43.470168 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:41:43.470168 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:41:43.470168 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:43.470168 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:41:43.470168 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:43.470168 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:41:43.470168 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:41:43.470168 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:43.470168 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:41:43.470594 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:43.470594 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:41:43.470794 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:43.470794 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:41:43.470794 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:41:43.470794 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:41:43.470794 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:41:43.470794 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:41:43.470794 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:43.470794 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:41:43.470794 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:41:43.470794 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:43.470794 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:41:43.470794 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:41:43.470794 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:43.470794 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:43.470794 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:41:43.470794 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:43.471237 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:41:43.471237 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:41:43.471237 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:43.471237 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:43.471237 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:43.471237 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:41:43.471237 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:43.471237 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:43.471237 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:43.471237 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:41:43.471237 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:43.471237 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:41:43.471237 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:41:43.471237 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:43.471237 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:41:43.471237 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:43.471237 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:41:43.471237 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:41:43.471237 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:43.471743 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:41:43.471743 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:43.471743 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:41:47.986354 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:41:47.986354 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:41:47.986354 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:41:47.987616 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:41:47.989789 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG dbcounter [-] [106123] Writing DB stats neutron:SELECT=239 {{(pid=106123) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} May 18 18:41:48.022521 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:41:48.022895 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:48.023754 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:41:48.023754 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:48.023754 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:41:48.023754 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:41:48.023754 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:48.023754 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:41:48.023754 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:48.023754 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:41:48.024915 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.038 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:48.024915 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:41:48.024915 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:48.024915 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:41:48.024915 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:41:48.024915 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:41:48.024915 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:41:48.024915 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:48.024915 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:48.024915 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:48.024915 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:41:48.024915 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:48.025549 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:48.025549 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:48.025549 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:41:48.025549 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:48.025549 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:41:48.025549 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:41:48.025549 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:48.025549 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:41:48.025549 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:48.025549 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:41:48.025549 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:41:48.025549 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:48.025549 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:41:48.026104 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:48.026104 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:41:48.026208 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:48.026208 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:41:48.026208 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:41:48.026208 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:41:48.026208 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:41:48.026208 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:41:48.026208 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:48.026208 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:41:48.026208 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:41:48.026208 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:48.026208 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:41:48.026208 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:41:48.026208 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:48.026208 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:48.026208 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:41:48.026208 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:48.026936 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:41:48.026936 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:41:48.026936 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:48.026936 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:48.026936 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:48.026936 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:41:48.026936 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:48.026936 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:48.026936 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:48.026936 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:41:48.026936 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:48.026936 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:41:48.026936 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:41:48.026936 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:48.026936 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:41:48.026936 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:48.026936 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:41:48.026936 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:41:48.026936 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:48.027796 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:41:48.027796 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:48.027796 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:41:48.447981 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:41:48.448881 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:41:48.449081 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:41:48.450297 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:41:48.450778 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:48.452433 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:41:48.452433 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:48.452433 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:41:48.452433 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:41:48.452433 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:48.452433 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:41:48.452433 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:48.452433 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:41:48.453569 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:48.453569 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:41:48.453569 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:48.453569 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:41:48.453569 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:41:48.453569 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:41:48.453569 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:41:48.453569 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:48.453569 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:48.453569 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:48.453569 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:41:48.453569 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:48.454147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:48.454147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:48.454147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:41:48.454147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:48.454147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:41:48.454147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:41:48.454147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:48.454147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:41:48.454147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:48.454147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:41:48.454147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:41:48.454147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:48.454147 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:41:48.454783 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:48.454783 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:41:48.455228 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:48.455228 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:41:48.455228 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:41:48.455228 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:41:48.455228 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:41:48.455228 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:41:48.455228 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:48.455228 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:41:48.455228 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:41:48.455228 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:48.455228 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:41:48.455228 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:41:48.455228 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:48.455228 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:48.455228 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:41:48.455228 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:48.455947 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:41:48.455947 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:41:48.455947 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:48.455947 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:48.455947 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:48.455947 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:41:48.455947 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:48.455947 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:48.455947 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:48.455947 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:41:48.455947 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:48.455947 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:41:48.455947 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:41:48.455947 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:48.455947 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:41:48.455947 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:48.455947 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:41:48.455947 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:41:48.455947 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:48.456928 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:41:48.456928 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:48.456928 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:41:48.456928 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:41:48.456928 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:41:48.456928 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:41:48.456928 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:41:48.459000 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:41:48.459000 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:48.459246 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:41:48.459246 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:48.459246 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:41:48.459246 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:41:48.459246 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:48.459246 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:41:48.459246 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:48.459246 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:41:48.460538 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:48.460538 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:41:48.460538 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:41:48.460538 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:41:48.460538 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:48.460538 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:41:48.460538 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:48.460538 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:41:48.460538 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:48.460538 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:41:48.460538 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:48.460538 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:41:48.460538 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:48.460538 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:41:48.460538 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:41:48.461049 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:48.461049 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:41:48.461049 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:48.461049 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:41:48.461049 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:41:48.461049 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:48.461049 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:41:48.461049 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:48.461049 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:41:48.461558 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:48.461558 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:41:48.461558 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:48.461558 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:41:48.461558 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:41:48.461558 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:41:48.461558 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:41:48.461558 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:41:48.461558 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:41:48.461558 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:41:48.461558 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:48.461558 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:48.461814 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:48.461814 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:41:48.461814 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:48.461814 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:48.461814 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:48.461814 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:41:48.461814 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:48.461814 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:41:48.461814 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:41:48.461814 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:48.461814 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:41:48.461814 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:48.461814 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:41:48.461814 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:41:48.462086 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:48.462086 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:41:48.462086 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:48.462086 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:41:48.462843 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:48.462843 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:41:48.462843 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:41:48.462843 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:41:48.462843 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:41:48.462843 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:41:48.462843 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:48.462843 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:41:48.462843 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:41:48.462843 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:48.462843 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:41:48.462843 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:41:48.462843 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:48.462843 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:48.462843 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:41:48.462843 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:48.463162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:41:48.463162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:41:48.463162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:41:48.463162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:41:48.463162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:41:48.463162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:48.463162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:48.463162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:48.463162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:41:48.463162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:48.463162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:48.463162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:48.463162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:41:48.463162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:48.463162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:41:48.463162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:41:48.463162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:48.463162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:41:48.463545 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:48.463545 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:41:48.463545 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:41:48.463545 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:48.463545 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:41:48.463545 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:48.463545 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:41:48.463545 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:41:48.463545 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:41:48.463545 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:41:48.463757 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:41:48.464475 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:41:48.464475 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:48.464910 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:41:48.464910 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:48.464910 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:41:48.464910 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:41:48.464910 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:48.464910 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:41:48.464910 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:48.464910 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:41:48.465749 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:48.465749 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:41:48.465749 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:41:48.465749 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:41:48.465749 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:48.465749 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:41:48.465749 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:48.465749 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:41:48.465749 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:48.465749 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:41:48.465749 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:48.465749 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:41:48.465749 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:48.465749 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:41:48.465749 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:41:48.466511 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:48.466511 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:41:48.466511 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:48.466511 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:41:48.466511 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:41:48.466511 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:48.466511 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:41:48.466511 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:48.466511 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:41:48.466946 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:48.466946 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:41:48.466946 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:48.466946 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:41:48.466946 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:41:48.466946 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:41:48.466946 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:41:48.466946 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:41:48.466946 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:41:48.466946 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:48.466946 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:48.467426 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:48.467426 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:41:48.467426 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:48.467426 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:48.467426 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:48.467426 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:41:48.467426 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:48.467426 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:41:48.467426 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:41:48.467426 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:48.467426 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:41:48.467426 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:48.467426 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:41:48.467426 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:41:48.468020 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:48.468020 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:41:48.468020 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:48.468020 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:41:48.468020 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:48.468020 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:41:48.468020 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:41:48.468020 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:41:48.468020 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:41:48.468020 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:41:48.468020 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:48.468020 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:41:48.468020 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:41:48.468020 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:48.468665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:41:48.468665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:41:48.468665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:48.468665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:48.468665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:41:48.468665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:48.468665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:41:48.468665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:41:48.468665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:41:48.468665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:41:48.468665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:48.468665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:48.468665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:48.468665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:41:48.468665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:48.468665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:48.468665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:48.468665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:41:48.468665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:48.469532 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:41:48.469532 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:41:48.469532 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:48.469532 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:41:48.469532 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:48.469532 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:41:48.469532 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:41:48.469532 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:48.469532 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:41:48.469532 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:48.469532 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:41:48.469532 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:41:48.469532 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:41:48.470085 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:41:48.470085 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:41:48.470085 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:41:48.470085 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:41:48.470085 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:48.470394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:41:48.470394 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:48.470394 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:41:48.470394 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:41:48.470394 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:48.470394 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:41:48.470394 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:48.470394 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:41:48.471274 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:48.471274 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:41:48.471274 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:48.471274 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:41:48.471274 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:41:48.471274 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:41:48.471274 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:41:48.471274 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:48.471274 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:48.471274 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:48.471274 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:41:48.471274 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:48.471851 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:48.471851 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:48.471851 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:41:48.471851 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:48.471851 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:41:48.471851 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:41:48.471851 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:48.471851 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:41:48.471851 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:48.471851 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:41:48.471851 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:41:48.471851 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:48.471851 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:41:48.472473 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:48.472473 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:41:48.472473 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:48.472473 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:41:48.472473 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:41:48.472473 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:41:48.472473 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:41:48.472473 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:41:48.472473 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:48.472473 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:41:48.472473 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:41:48.472473 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:48.472473 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:41:48.472473 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:41:48.472473 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:48.473137 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:48.473137 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:41:48.473137 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:48.473137 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:41:48.473137 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:41:48.473137 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:48.473137 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:48.473137 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:48.473137 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:41:48.473137 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:48.473137 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:48.473137 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:48.473137 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:41:48.473137 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:48.473137 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:41:48.473137 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:41:48.473137 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:48.473137 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:41:48.473137 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:48.474008 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:41:48.474008 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:41:48.474008 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:48.474008 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:41:48.474008 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:48.474008 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:41:52.986025 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:41:52.986888 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:41:52.987083 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:41:52.988166 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:41:53.023585 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:41:53.024012 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:53.025013 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:41:53.025013 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:53.025013 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:41:53.025013 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:41:53.025013 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:53.025013 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:41:53.025013 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:53.025013 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:41:53.026050 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.038 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:53.026050 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:41:53.026050 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:53.026050 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:41:53.026050 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:41:53.026050 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:41:53.026050 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:41:53.026050 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:53.026050 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:53.026050 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:53.026050 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:41:53.026050 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:53.026619 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:53.026619 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:53.026619 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:41:53.026619 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:53.026619 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:41:53.026619 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:41:53.026619 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:53.026619 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:41:53.026619 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:53.026619 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:41:53.026619 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:41:53.026619 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:53.026619 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:41:53.027151 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:53.027151 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:41:53.027151 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:53.027151 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:41:53.027151 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:41:53.027151 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:41:53.027151 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:41:53.027151 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:41:53.027151 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:53.027151 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:41:53.027151 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:41:53.027151 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:53.027151 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:41:53.027151 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:41:53.027151 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:53.027829 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:53.027829 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:41:53.027829 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:53.027829 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:41:53.027829 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:41:53.027829 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:53.027829 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:53.027829 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:53.027829 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:41:53.027829 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:53.027829 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:53.027829 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:53.027829 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:41:53.027829 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:53.027829 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:41:53.027829 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:41:53.027829 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:53.027829 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:41:53.027829 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:53.028655 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:41:53.028655 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:41:53.028655 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:53.028655 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:41:53.028655 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:53.028655 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:41:53.448642 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:41:53.449162 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:41:53.449268 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:41:53.450219 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:41:53.450639 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:53.451530 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:41:53.451530 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:53.451530 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:41:53.451530 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:41:53.451530 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:53.451530 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:41:53.451530 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:53.451530 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:41:53.452768 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:53.452768 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:41:53.452768 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:53.452768 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:41:53.452768 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:41:53.452768 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:41:53.452768 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:41:53.452768 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:53.452768 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:53.452768 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:53.452768 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:41:53.452768 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:53.453580 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:53.453580 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:53.453580 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:41:53.453580 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:53.453580 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:41:53.453580 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:41:53.453580 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:53.453580 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:41:53.453580 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:53.453580 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:41:53.453580 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:41:53.453580 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:53.453580 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:41:53.454072 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:53.454072 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:41:53.454072 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:53.454072 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:41:53.454072 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:41:53.454072 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:41:53.454072 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:41:53.454072 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:41:53.454072 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:53.454072 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:41:53.454072 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:41:53.454072 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:53.454072 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:41:53.454072 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:41:53.454072 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:53.454665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:53.454665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:41:53.454665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:53.454665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:41:53.454665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:41:53.454665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:53.454665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:53.454665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:53.454665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:41:53.454665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:53.454665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:53.454665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:53.454665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:41:53.454665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:53.454665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:41:53.454665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:41:53.454665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:53.454665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:41:53.454665 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:53.455393 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:41:53.455393 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:41:53.455393 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:53.455393 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:41:53.455393 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:53.455393 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:41:53.456435 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:41:53.456953 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:41:53.456953 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:41:53.456953 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:41:53.457529 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:41:53.457947 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:53.458740 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:41:53.458740 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:53.458740 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:41:53.458740 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:41:53.458740 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:53.458740 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:41:53.458740 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:53.458740 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:41:53.459509 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:53.459509 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:41:53.459509 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:41:53.459509 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:41:53.459509 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:53.459509 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:41:53.459509 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:53.459509 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:41:53.459509 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:53.459509 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:41:53.459509 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:53.459509 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:41:53.459509 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:53.459509 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:41:53.459509 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:41:53.460235 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:53.460235 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:41:53.460235 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:53.460235 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:41:53.460235 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:41:53.460235 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:53.460235 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:41:53.460235 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:53.460235 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:41:53.460667 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:53.460667 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:41:53.460667 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:53.460667 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:41:53.460667 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:41:53.460667 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:41:53.460667 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:41:53.460667 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:41:53.460667 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:41:53.460667 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:41:53.460667 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:53.460667 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:53.461375 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:53.461375 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:41:53.461375 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:53.461375 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:53.461375 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:53.461375 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:41:53.461375 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:53.461375 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:41:53.461375 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:41:53.461375 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:53.461375 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:41:53.461375 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:53.461375 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:41:53.461375 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:41:53.462129 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:53.462129 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:41:53.462129 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:53.462129 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:41:53.462129 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:53.462129 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:41:53.462129 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:41:53.462129 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:41:53.462129 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:41:53.462129 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:41:53.462129 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:53.462129 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:41:53.462129 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:41:53.462129 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:53.463026 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:41:53.463026 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:41:53.463026 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:53.463026 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:53.463026 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:41:53.463026 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:53.463026 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:41:53.463026 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:41:53.463026 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:41:53.463026 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:41:53.463026 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:41:53.463026 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:53.463026 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:53.463026 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:53.463026 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:41:53.463026 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:53.463026 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:53.463026 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:53.463026 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:41:53.463997 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:53.463997 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:41:53.463997 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:41:53.463997 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:53.463997 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:41:53.463997 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:53.463997 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:41:53.463997 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:41:53.463997 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:53.463997 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:41:53.463997 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:53.463997 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:41:53.463997 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:41:53.463997 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:41:53.464906 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:41:53.464906 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:41:53.464906 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:41:53.464906 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:53.465220 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:41:53.465220 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:53.465220 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:41:53.465220 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:41:53.465220 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:53.465220 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:41:53.465220 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:53.465220 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:41:53.465753 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:53.465753 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:41:53.465753 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:41:53.465753 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:41:53.465753 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:53.465753 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:41:53.465753 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:53.465753 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:41:53.465753 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:53.465753 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:41:53.465753 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:53.465753 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:41:53.465753 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:53.465753 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:41:53.465753 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:41:53.466466 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:53.466466 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:41:53.466466 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:53.466466 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:41:53.466466 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:41:53.466466 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:53.466466 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:41:53.466466 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:53.466466 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:41:53.466753 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:53.466753 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:41:53.466753 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:53.466753 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:41:53.466753 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:41:53.466753 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:41:53.466753 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:41:53.466753 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:41:53.466753 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:41:53.466753 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:53.466753 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:53.467203 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:53.467203 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:41:53.467203 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:53.467203 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:53.467203 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:53.467203 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:41:53.467203 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:53.467203 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:41:53.467203 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:41:53.467203 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:53.467203 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:41:53.467203 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:53.467203 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:41:53.467203 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:41:53.467746 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:53.467746 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:41:53.467746 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:53.467746 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:41:53.467746 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:53.467746 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:41:53.467746 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:41:53.467746 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:41:53.467746 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:41:53.467746 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:41:53.467746 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:53.467746 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:41:53.467746 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:41:53.467746 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:53.468235 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:41:53.468235 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:41:53.468235 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:53.468235 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:53.468235 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:41:53.468235 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:53.468235 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:41:53.468235 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:41:53.468235 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:41:53.468235 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:41:53.468235 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:53.468235 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:53.468235 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:53.468235 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:41:53.468235 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:53.468235 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:53.468235 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:53.468235 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:41:53.468235 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:53.468798 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:41:53.468798 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:41:53.468798 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:53.468798 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:41:53.468798 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:53.468798 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:41:53.468798 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:41:53.468798 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:53.468798 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:41:53.468798 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:53.468798 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:41:53.468798 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:41:53.469155 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:41:53.469155 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:41:53.469155 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:41:53.469503 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:41:53.469702 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:41:53.469807 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:53.470351 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:41:53.470351 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:53.470351 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:41:53.470351 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:41:53.470351 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:53.470351 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:41:53.470351 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:53.470351 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:41:53.471143 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:53.471143 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:41:53.471143 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:53.471143 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:41:53.471143 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:41:53.471143 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:41:53.471143 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:41:53.471143 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:53.471143 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:53.471143 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:53.471143 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:41:53.471143 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:53.471633 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:53.471633 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:53.471633 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:41:53.471633 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:53.471633 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:41:53.471633 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:41:53.471633 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:53.471633 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:41:53.471633 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:53.471633 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:41:53.471633 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:41:53.471633 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:53.471633 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:41:53.472015 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:53.472015 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:41:53.472174 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:53.472174 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:41:53.472174 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:41:53.472174 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:41:53.472174 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:41:53.472174 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:41:53.472174 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:53.472174 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:41:53.472174 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:41:53.472174 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:53.472174 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:41:53.472174 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:41:53.472174 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:53.472174 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:53.472174 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:41:53.472174 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:53.472859 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:41:53.472859 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:41:53.472859 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:53.472859 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:53.472859 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:53.472859 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:41:53.472859 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:53.472859 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:53.472859 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:53.472859 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:41:53.472859 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:53.472859 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:41:53.472859 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:41:53.472859 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:53.472859 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:41:53.472859 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:53.472859 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:41:53.472859 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:41:53.472859 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:53.473484 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:41:53.473484 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:53.473484 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:41:57.986759 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:41:57.987543 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:41:57.987543 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:41:57.988230 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:41:58.024992 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:41:58.025368 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:58.026221 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:41:58.026221 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:58.026221 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:41:58.026221 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:41:58.026221 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:58.026221 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:41:58.026221 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:58.026221 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:41:58.027238 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.039 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:58.027238 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:41:58.027238 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:58.027238 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:41:58.027238 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:41:58.027238 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:41:58.027238 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:41:58.027238 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:58.027238 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:58.027238 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:58.027238 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:41:58.027238 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:58.027924 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:58.027924 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:58.027924 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:41:58.027924 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:58.027924 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:41:58.027924 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:41:58.027924 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:58.027924 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:41:58.027924 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:58.027924 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:41:58.027924 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:41:58.027924 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:58.027924 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:41:58.029136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:58.029136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:41:58.029136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:58.029136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:41:58.029136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:41:58.029136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:41:58.029136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:41:58.029136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:41:58.029136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:58.029136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:41:58.029136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:41:58.029136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:58.029136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:41:58.029136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:41:58.029136 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:58.029780 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:58.029780 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:41:58.029780 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:58.029780 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:41:58.029780 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:41:58.029780 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:58.029780 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:58.029780 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:58.029780 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:41:58.029780 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:58.029780 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:58.029780 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:58.029780 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:41:58.029780 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:58.029780 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:41:58.029780 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:41:58.029780 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:58.029780 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:41:58.029780 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:58.031843 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:41:58.031843 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:41:58.031843 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:58.031843 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:41:58.031843 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:58.031843 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:41:58.449987 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:41:58.450835 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:41:58.451047 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:41:58.452143 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:41:58.452505 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:58.453470 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:41:58.453470 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:58.453470 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:41:58.453470 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:41:58.453470 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:58.453470 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:41:58.453470 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:58.453470 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:41:58.455246 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:58.455246 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:41:58.455246 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:58.455246 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:41:58.455246 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:41:58.455246 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:41:58.455246 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:41:58.455246 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:58.455246 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:58.455246 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:58.455246 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:41:58.455246 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:58.455619 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:58.455619 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:58.455619 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:41:58.455619 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:58.455619 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:41:58.455619 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:41:58.455619 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:58.455619 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:41:58.455619 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:58.455619 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:41:58.455619 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:41:58.455619 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:58.455619 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:41:58.455971 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:58.455971 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:41:58.457068 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:58.457068 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:41:58.457068 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:41:58.457068 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:41:58.457068 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:41:58.457068 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:41:58.457068 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:58.457068 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:41:58.457068 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:41:58.457068 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:58.457068 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:41:58.457068 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:41:58.457068 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:58.457068 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:58.457068 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:41:58.457068 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:58.457687 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:41:58.457687 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:41:58.457687 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:58.457687 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:58.457687 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:58.457687 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:41:58.457687 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:58.457687 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:58.457687 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:58.457687 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:41:58.457687 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:58.457687 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:41:58.457687 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:41:58.457687 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:58.457687 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:41:58.457687 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:58.457687 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:41:58.457687 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:41:58.457687 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:58.458129 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:41:58.458129 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:58.458129 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:41:58.458129 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:41:58.458129 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:41:58.458129 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:41:58.458129 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:41:58.458672 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:41:58.458892 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:58.459724 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:41:58.459724 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:58.459724 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:41:58.459724 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:41:58.459724 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:58.459724 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:41:58.459724 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:58.459724 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:41:58.461002 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:58.461002 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:41:58.461002 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:41:58.461002 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:41:58.461002 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:58.461002 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:41:58.461002 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:58.461002 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:41:58.461002 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:58.461002 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:41:58.461002 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:58.461002 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:41:58.461002 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:58.461002 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:41:58.461002 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:41:58.461753 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:58.461753 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:41:58.461753 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:58.461753 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:41:58.461753 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:41:58.461753 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:58.461753 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:41:58.461753 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:58.461753 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:41:58.462338 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:58.462338 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:41:58.462338 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:58.462338 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:41:58.462338 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:41:58.462338 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:41:58.462338 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:41:58.462338 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:41:58.462338 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:41:58.462338 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:41:58.462338 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:58.462338 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:58.462976 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:58.462976 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:41:58.462976 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:58.462976 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:58.462976 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:58.462976 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:41:58.462976 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:58.462976 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:41:58.462976 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:41:58.462976 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:58.462976 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:41:58.462976 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:58.462976 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:41:58.462976 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:41:58.463713 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:58.463713 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:41:58.463713 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:58.463713 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:41:58.464833 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:58.464833 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:41:58.464833 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:41:58.464833 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:41:58.464833 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:41:58.464833 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:41:58.464833 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:58.464833 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:41:58.464833 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:41:58.464833 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:58.464833 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:41:58.464833 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:41:58.464833 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:58.464833 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:58.464833 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:41:58.464833 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:58.465550 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:41:58.465550 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:41:58.465550 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:41:58.465550 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:41:58.465550 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:41:58.465550 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:58.465550 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:58.465550 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:58.465550 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:41:58.465550 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:58.465550 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:58.465550 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:58.465550 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:41:58.465550 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:58.465550 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:41:58.465550 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:41:58.465550 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:58.465550 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:41:58.466252 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:58.466252 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:41:58.466252 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:41:58.466252 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:58.466252 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:41:58.466252 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:58.466252 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:41:58.466252 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:41:58.466252 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:41:58.466252 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:41:58.466522 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:41:58.466522 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:41:58.466522 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:58.467038 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:41:58.467038 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:58.467038 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:41:58.467038 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:41:58.467038 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:58.467038 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:41:58.467038 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:58.467038 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:41:58.467620 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:58.467620 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:41:58.467620 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:41:58.467620 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:41:58.467620 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:58.467620 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:41:58.467620 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:58.467620 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:41:58.467620 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:58.467620 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:41:58.467620 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:58.467620 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:41:58.467620 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:58.467620 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:41:58.467620 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:41:58.468008 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:58.468008 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:41:58.468008 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:58.468008 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:41:58.468008 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:41:58.468008 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:58.468008 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:41:58.468008 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:58.468008 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:41:58.468381 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:58.468381 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:41:58.468381 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:58.468381 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:41:58.468381 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:41:58.468381 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:41:58.468381 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:41:58.468381 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:41:58.468381 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:41:58.468381 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:58.468381 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:58.468888 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:58.468888 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:41:58.468888 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:58.468888 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:58.468888 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:58.468888 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:41:58.468888 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:58.468888 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:41:58.468888 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:41:58.468888 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:58.468888 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:41:58.468888 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:58.468888 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:41:58.468888 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:41:58.469550 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:58.469550 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:41:58.469550 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:58.469550 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:41:58.469550 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:58.469550 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:41:58.469550 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:41:58.469550 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:41:58.469550 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:41:58.469550 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:41:58.469550 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:58.469550 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:41:58.469550 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:41:58.469550 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:58.470002 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:41:58.470002 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:41:58.470002 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:58.470002 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:58.470002 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:41:58.470002 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:58.470002 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:41:58.470002 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:41:58.470002 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:41:58.470002 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:41:58.470002 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:58.470002 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:58.470002 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:58.470002 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:41:58.470002 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:58.470002 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:58.470002 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:58.470002 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:41:58.470002 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:58.470439 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:41:58.470439 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:41:58.470439 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:58.470439 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:41:58.470439 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:58.470439 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:41:58.470439 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:41:58.470439 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:58.470439 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:41:58.470439 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:58.470439 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:41:58.470439 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:41:58.470439 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:41:58.470751 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:41:58.470751 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:41:58.470751 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:41:58.470751 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:41:58.470860 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:58.471457 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:41:58.471457 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:58.471457 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:41:58.471457 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:41:58.471457 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:58.471457 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:41:58.471457 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:58.471457 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:41:58.472094 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:58.472094 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:41:58.472094 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:58.472094 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:41:58.472094 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:41:58.472094 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:41:58.472094 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:41:58.472094 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:58.472094 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:58.472094 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:58.472094 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:41:58.472094 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:58.472424 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:41:58.472424 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:58.472424 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:41:58.472424 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:58.472424 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:41:58.472424 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:41:58.472424 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:58.472424 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:41:58.472424 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:58.472424 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:41:58.472424 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:41:58.472424 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:58.472424 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:41:58.472710 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:58.472710 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:41:58.472986 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:58.472986 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:41:58.472986 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:41:58.472986 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:41:58.472986 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:41:58.472986 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:41:58.472986 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:58.472986 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:41:58.472986 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:41:58.472986 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:58.472986 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:41:58.472986 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:41:58.472986 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:41:58.472986 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:41:58.472986 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:41:58.472986 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:41:58.473394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:41:58.473394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:41:58.473394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:58.473394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:58.473394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:41:58.473394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:41:58.473394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:41:58.473394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:41:58.473394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:41:58.473394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:41:58.473394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:41:58.473394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:41:58.473394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:41:58.473394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:41:58.473394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:41:58.473394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:41:58.473394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:41:58.473394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:41:58.473394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:41:58.473828 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:41:58.473828 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:41:58.473828 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:42:02.987310 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:42:02.991382 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:42:02.991382 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:42:02.991382 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:42:03.019776 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:42:03.020055 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:03.022394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:42:03.022394 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:03.022394 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:42:03.022394 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:42:03.022394 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:03.022394 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:42:03.022394 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:03.022394 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:42:03.022394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.032 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:03.022394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:42:03.022394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:42:03.022394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:42:03.022394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:42:03.022394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:42:03.022394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:42:03.022743 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:03.022743 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:42:03.022743 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:42:03.022743 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:42:03.022743 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:03.022743 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:42:03.022743 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:42:03.022743 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:42:03.022743 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:42:03.022743 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:42:03.022743 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:42:03.022743 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:42:03.022743 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:42:03.022743 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:03.023000 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:42:03.023000 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:42:03.023000 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:03.023000 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:42:03.023000 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:03.023000 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:42:03.023000 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:03.023000 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:42:03.023000 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:42:03.023000 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:42:03.023000 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:42:03.023000 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:42:03.023000 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:42:03.023000 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:42:03.023278 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:42:03.023278 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:42:03.023278 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:42:03.023278 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:42:03.023278 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:42:03.023278 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:42:03.023278 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:42:03.023278 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:42:03.023278 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:42:03.023278 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:42:03.023278 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:03.023278 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:42:03.023278 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:42:03.023278 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:42:03.023278 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:03.023278 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:42:03.023278 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:42:03.023278 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:42:03.023278 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:42:03.023278 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:42:03.023278 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:42:03.023680 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:42:03.023680 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:42:03.023680 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:03.023680 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:42:03.023680 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:42:03.023680 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:03.023680 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:42:03.023680 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:03.023680 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:42:03.450976 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:42:03.452373 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:42:03.452373 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:42:03.453029 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:42:03.453431 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:03.457356 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:42:03.457356 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:03.457356 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:42:03.457356 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:42:03.457356 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:03.457356 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:42:03.457356 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:03.457356 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:42:03.457356 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:03.457356 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:42:03.457356 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:42:03.457356 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:42:03.457356 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:42:03.457356 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:42:03.457356 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:42:03.458361 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:03.458361 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:42:03.458361 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:42:03.458361 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:42:03.458361 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:03.458361 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:42:03.458361 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:42:03.458361 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:42:03.458361 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:42:03.458361 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:42:03.458361 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:42:03.458361 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:42:03.458361 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:42:03.458361 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:03.459087 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:42:03.459087 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:42:03.459087 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:03.459087 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:42:03.459087 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:03.459087 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:42:03.459906 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:03.459906 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:42:03.459906 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:42:03.459906 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:42:03.459906 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:42:03.459906 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:42:03.459906 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:42:03.459906 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:42:03.459906 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:42:03.459906 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:42:03.459906 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:42:03.459906 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:42:03.459906 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:42:03.459906 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:42:03.459906 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:42:03.459906 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:42:03.460682 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:42:03.460682 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:42:03.460682 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:03.460682 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:42:03.460682 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:42:03.460682 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:42:03.460682 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:03.460682 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:42:03.460682 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:42:03.460682 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:42:03.460682 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:42:03.460682 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:42:03.460682 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:42:03.460682 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:42:03.460682 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:42:03.460682 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:03.460682 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:42:03.460682 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:42:03.460682 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:03.461619 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:42:03.461619 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:03.461619 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:42:03.462018 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:42:03.462815 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:42:03.462815 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:42:03.462815 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:42:03.463131 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:42:03.463335 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:03.465733 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:42:03.465733 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:03.465733 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:42:03.465733 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:42:03.465733 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:03.465733 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:42:03.465733 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:03.465733 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:42:03.465733 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:03.465733 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:42:03.465733 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:42:03.465733 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:42:03.465733 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:03.465733 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:42:03.465733 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:42:03.465733 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:42:03.465733 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:03.465733 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:42:03.466152 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:42:03.466152 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:42:03.466152 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:42:03.466152 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:42:03.466152 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:42:03.466152 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:42:03.466152 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:42:03.466152 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:03.466152 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:42:03.466152 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:42:03.466152 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:03.466152 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:42:03.466152 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:03.466152 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:42:03.466152 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:03.466497 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:42:03.466497 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:42:03.466497 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:42:03.466497 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:42:03.466497 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:42:03.466497 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:42:03.466497 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:42:03.466497 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:42:03.466497 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:42:03.466497 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:03.466497 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:42:03.466497 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:42:03.466497 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:42:03.466743 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:03.466743 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:42:03.466743 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:42:03.466743 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:42:03.466743 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:42:03.466743 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:42:03.466743 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:42:03.466743 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:42:03.466743 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:42:03.466743 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:03.466743 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:42:03.466743 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:42:03.466743 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:03.466743 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:42:03.467019 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:03.467019 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:42:03.467019 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:03.467019 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:42:03.467019 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:42:03.467019 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:42:03.467019 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:42:03.467019 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:42:03.467019 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:42:03.467019 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:42:03.467019 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:42:03.467019 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:42:03.467019 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:42:03.467019 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:42:03.467019 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:42:03.467312 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:42:03.467312 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:42:03.467312 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:42:03.467312 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:42:03.467312 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:42:03.467312 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:42:03.467312 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:42:03.467312 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:42:03.467312 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:03.467312 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:42:03.467312 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:42:03.467312 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:42:03.467312 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:03.467312 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:42:03.467312 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:42:03.467312 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:42:03.467312 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:42:03.467312 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:42:03.467312 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:42:03.467710 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:42:03.467710 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:42:03.467710 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:03.467710 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:42:03.467710 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:42:03.467710 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:03.467710 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:42:03.467710 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:03.467710 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:42:03.467710 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:42:03.468085 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:42:03.468238 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:42:03.468414 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:42:03.468867 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:42:03.469098 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:03.469638 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:42:03.469638 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:03.469638 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:42:03.469638 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:42:03.469638 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:03.469638 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:42:03.469638 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:03.469638 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:42:03.470094 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:42:03.473356 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:03.473356 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:42:03.473356 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:42:03.473356 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:42:03.473356 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:03.473356 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:42:03.473356 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:42:03.473356 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:42:03.473356 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:03.473356 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:42:03.473356 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:42:03.473356 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:42:03.473356 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:42:03.473356 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:42:03.473356 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:42:03.473901 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:42:03.473901 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:42:03.473901 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:03.473901 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:42:03.473901 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:42:03.473901 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:03.473901 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:42:03.473901 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:03.473901 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:42:03.473901 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:03.473901 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:42:03.473901 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:42:03.473901 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:42:03.473901 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:42:03.474316 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:42:03.474316 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:42:03.474316 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:42:03.474316 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:42:03.474316 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:03.474316 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:42:03.474316 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:42:03.474316 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:42:03.474316 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:03.474316 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:42:03.474316 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:42:03.474316 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:42:03.474316 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:42:03.474696 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:42:03.474696 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:42:03.474696 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:42:03.474696 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:42:03.474696 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:03.474696 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:42:03.474696 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:42:03.474696 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:03.474696 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:42:03.474696 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:03.474696 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:42:03.474696 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:03.475047 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:42:03.475047 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:42:03.475047 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:42:03.475047 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:42:03.475047 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:42:03.475047 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:42:03.475047 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:42:03.475047 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:42:03.475047 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:42:03.475047 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:42:03.475047 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:42:03.475047 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:42:03.475047 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:42:03.475047 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:42:03.475047 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:42:03.475047 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:42:03.475047 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:42:03.475047 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:42:03.475047 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:42:03.475636 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:03.475636 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:42:03.475636 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:42:03.475636 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:42:03.475636 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:03.475636 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:42:03.475636 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:42:03.475636 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:42:03.475636 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:42:03.475636 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:42:03.475636 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:42:03.475636 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:42:03.475636 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:42:03.475636 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:03.475636 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:42:03.475636 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:42:03.475636 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:03.475636 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:42:03.475636 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:03.475636 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:42:03.476174 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:42:03.476174 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:42:03.476502 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:42:03.476969 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:42:03.477312 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:42:03.477581 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:03.479687 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:42:03.479687 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:03.479687 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:42:03.479687 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:42:03.479687 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:03.479687 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:42:03.479687 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:03.479687 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:42:03.479687 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:03.479687 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:42:03.479687 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:42:03.479687 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:42:03.479687 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:42:03.479687 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:42:03.479687 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:42:03.480019 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:03.480019 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:42:03.480019 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:42:03.480019 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:42:03.480019 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:03.480019 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:42:03.480019 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:42:03.480019 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:42:03.480019 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:42:03.480019 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:42:03.480019 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:42:03.480019 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:42:03.480019 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:42:03.480019 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:03.480310 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:42:03.480310 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:42:03.480310 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:03.480310 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:42:03.480310 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:03.480310 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:42:03.480310 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:03.480310 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:42:03.480310 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:42:03.480310 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:42:03.480310 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:42:03.480310 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:42:03.480310 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:42:03.480310 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:42:03.480602 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:42:03.480602 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:42:03.480602 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:42:03.480602 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:42:03.480602 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:42:03.480602 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:42:03.480602 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:42:03.480602 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:42:03.480602 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:42:03.480602 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:42:03.480602 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:03.480602 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:42:03.480602 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:42:03.480602 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:42:03.480602 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:03.480602 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:42:03.480602 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:42:03.480602 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:42:03.480602 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:42:03.480602 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:42:03.481015 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:42:03.481015 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:42:03.481015 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:42:03.481015 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:03.481015 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:42:03.481015 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:42:03.481015 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:03.481015 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:42:03.481015 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:03.481015 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:42:07.988990 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:42:07.989919 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:42:07.989919 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:42:07.991579 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:42:08.025471 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:42:08.025471 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:08.026303 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:42:08.026303 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:08.026303 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:42:08.026303 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:42:08.026303 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:08.026303 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:42:08.026303 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:08.026303 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:42:08.027257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.036 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:08.027257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:42:08.027257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:42:08.027257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:42:08.027257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:42:08.027257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:42:08.027257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:42:08.027257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:08.027257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:42:08.027257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:42:08.027257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:42:08.027257 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:08.027997 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:42:08.027997 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:42:08.027997 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:42:08.027997 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:42:08.027997 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:42:08.027997 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:42:08.027997 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:42:08.027997 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:42:08.027997 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:08.027997 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:42:08.027997 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:42:08.027997 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:08.027997 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:42:08.028678 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:08.028678 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:42:08.028678 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:08.028678 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:42:08.028678 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:42:08.028678 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:42:08.028678 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:42:08.028678 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:42:08.028678 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:42:08.028678 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:42:08.028678 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:42:08.028678 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:42:08.028678 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:42:08.028678 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:42:08.028678 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:42:08.029468 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:42:08.029468 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:42:08.029468 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:42:08.029468 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:42:08.029468 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:42:08.029468 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:08.029468 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:42:08.029468 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:42:08.029468 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:42:08.029468 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:08.029468 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:42:08.029468 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:42:08.029468 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:42:08.029468 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:42:08.029468 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:42:08.029468 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:42:08.029468 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:42:08.029468 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:42:08.029468 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:08.030445 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:42:08.030445 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:42:08.030445 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:08.030445 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:42:08.030445 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:08.030445 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:42:08.451884 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:42:08.453095 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:42:08.453301 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:42:08.454620 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:42:08.455052 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:08.456066 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:42:08.456066 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:08.456066 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:42:08.456066 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:42:08.456066 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:08.456066 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:42:08.456066 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:08.456066 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:42:08.458070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:08.458070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:42:08.458070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:42:08.458070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:42:08.458070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:42:08.458070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:42:08.458070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:42:08.458070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:08.458070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:42:08.458070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:42:08.458070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:42:08.458070 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:08.458728 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:42:08.458728 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:42:08.458728 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:42:08.458728 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:42:08.458728 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:42:08.458728 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:42:08.458728 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:42:08.458728 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:42:08.458728 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:08.458728 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:42:08.458728 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:42:08.458728 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:08.458728 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:42:08.459308 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:08.459308 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:42:08.459937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:08.459937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:42:08.459937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:42:08.459937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:42:08.459937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:42:08.459937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:42:08.459937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:42:08.459937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:42:08.459937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:42:08.459937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:42:08.459937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:42:08.459937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:42:08.459937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:42:08.459937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:42:08.459937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:42:08.459937 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:42:08.460391 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:42:08.460391 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:42:08.460391 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:08.460391 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:42:08.460391 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:42:08.460391 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:42:08.460391 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:08.460391 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:42:08.460391 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:42:08.460391 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:42:08.460391 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:42:08.460391 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:42:08.460391 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:42:08.460391 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:42:08.460391 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:42:08.460391 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:08.460391 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:42:08.460391 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:42:08.460391 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:08.460862 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:42:08.460862 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:08.460862 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:42:08.462621 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:42:08.464142 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:42:08.464265 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:42:08.464413 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:42:08.464827 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:42:08.465059 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:08.465681 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:42:08.465681 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:08.465681 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:42:08.465681 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:42:08.465681 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:08.465681 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:42:08.465681 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:08.465681 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:42:08.466478 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:08.466478 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:42:08.466478 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:42:08.466478 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:42:08.466478 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:08.466478 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:42:08.466478 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:42:08.466478 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:42:08.466478 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:08.466478 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:42:08.466478 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:42:08.466478 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:42:08.466478 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:42:08.466478 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:42:08.466478 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:42:08.466917 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:42:08.466917 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:42:08.466917 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:08.466917 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:42:08.466917 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:42:08.466917 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:08.466917 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:42:08.466917 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:08.466917 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:42:08.467337 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:08.467337 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:42:08.467337 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:42:08.467337 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:42:08.467337 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:42:08.467337 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:42:08.467337 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:42:08.467337 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:42:08.467337 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:42:08.467337 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:42:08.467337 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:08.467337 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:42:08.467749 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:42:08.467749 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:42:08.467749 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:08.467749 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:42:08.467749 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:42:08.467749 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:42:08.467749 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:42:08.467749 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:42:08.467749 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:42:08.467749 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:42:08.467749 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:42:08.467749 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:08.467749 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:42:08.467749 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:42:08.468097 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:08.468097 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:42:08.468097 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:08.468097 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:42:08.468346 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:08.468346 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:42:08.468346 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:42:08.468346 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:42:08.468346 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:42:08.468346 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:42:08.468346 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:42:08.468346 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:42:08.468346 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:42:08.468346 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:42:08.468346 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:42:08.468346 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:42:08.468346 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:42:08.468346 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:42:08.468346 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:42:08.468346 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:42:08.468780 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:42:08.468780 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:42:08.468780 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:42:08.468780 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:42:08.468780 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:42:08.468780 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:08.468780 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:42:08.468780 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:42:08.468780 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:42:08.468780 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:08.468780 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:42:08.468780 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:42:08.468780 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:42:08.468780 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:42:08.468780 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:42:08.468780 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:42:08.468780 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:42:08.468780 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:42:08.469269 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:08.469269 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:42:08.469269 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:42:08.469269 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:08.469269 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:42:08.469269 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:08.469269 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:42:08.469269 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:42:08.469269 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:42:08.469269 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:42:08.469647 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:42:08.469927 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:42:08.470305 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:08.471097 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:42:08.471097 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:08.471097 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:42:08.471097 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:42:08.471097 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:08.471097 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:42:08.471097 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:08.471097 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:42:08.471899 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:08.471899 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:42:08.471899 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:42:08.471899 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:42:08.471899 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:08.471899 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:42:08.471899 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:42:08.471899 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:42:08.471899 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:08.471899 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:42:08.471899 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:42:08.471899 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:42:08.471899 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:42:08.471899 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:42:08.471899 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:42:08.472296 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:42:08.472296 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:42:08.472296 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:08.472296 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:42:08.472296 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:42:08.472296 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:08.472296 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:42:08.472296 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:08.472296 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:42:08.472773 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:08.472773 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:42:08.472773 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:42:08.472773 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:42:08.472773 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:42:08.472773 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:42:08.472773 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:42:08.472773 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:42:08.472773 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:42:08.472773 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:08.472773 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:42:08.473098 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:42:08.473098 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:42:08.473098 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:08.473098 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:42:08.473098 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:42:08.473098 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:42:08.473098 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:42:08.473098 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:42:08.473098 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:42:08.473098 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:42:08.473098 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:42:08.473098 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:08.473098 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:42:08.473098 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:42:08.473509 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:08.473509 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:42:08.473509 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:08.473509 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:42:08.474026 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:08.474026 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:42:08.474026 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:42:08.474026 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:42:08.474026 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:42:08.474026 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:42:08.474026 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:42:08.474026 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:42:08.474026 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:42:08.474026 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:42:08.474026 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:42:08.474026 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:42:08.474026 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:42:08.474026 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:42:08.474026 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:42:08.474026 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:42:08.474026 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:42:08.474567 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:42:08.474567 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:42:08.474567 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:42:08.474567 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:08.474567 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:42:08.474567 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:42:08.474567 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:42:08.474567 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:08.474567 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:42:08.474567 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:42:08.474567 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:42:08.474567 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:42:08.474567 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:42:08.474567 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:42:08.474567 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:42:08.474567 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:42:08.474567 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:08.474567 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:42:08.474567 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:42:08.475051 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:08.475051 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:42:08.475051 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:08.475051 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:42:08.475051 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:42:08.475051 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:42:08.475051 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:42:08.475051 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:42:08.475357 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:42:08.475704 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:42:08.475920 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:08.476684 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:42:08.476684 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:08.476684 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:42:08.476684 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:42:08.476684 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:08.476684 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:42:08.476684 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:08.476684 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:42:08.477549 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:08.477549 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:42:08.477549 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:42:08.477549 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:42:08.477549 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:42:08.477549 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:42:08.477549 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:42:08.477549 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:08.477549 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:42:08.477549 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:42:08.477549 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:42:08.477549 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:08.477896 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:42:08.477896 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:42:08.477896 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:42:08.477896 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:42:08.477896 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:42:08.477896 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:42:08.477896 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:42:08.477896 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:42:08.477896 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:08.477896 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:42:08.477896 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:42:08.477896 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:08.477896 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:42:08.478226 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:08.478226 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:42:08.478518 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:08.478518 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:42:08.478518 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:42:08.478518 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:42:08.478518 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:42:08.478518 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:42:08.478518 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:42:08.478518 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:42:08.478518 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:42:08.478518 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:42:08.478518 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:42:08.478518 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:42:08.478518 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:42:08.478518 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:42:08.478518 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:42:08.478518 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:42:08.478970 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:42:08.478970 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:42:08.478970 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:08.478970 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:42:08.478970 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:42:08.478970 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:42:08.478970 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:08.478970 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:42:08.478970 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:42:08.478970 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:42:08.478970 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:42:08.478970 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:42:08.478970 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:42:08.478970 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:42:08.478970 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:42:08.478970 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:08.478970 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:42:08.478970 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:42:08.478970 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:08.479571 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:42:08.479571 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:08.479571 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:42:12.989550 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:42:12.990476 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:42:12.990725 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:42:12.992060 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:42:13.029039 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:42:13.029477 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:13.030264 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:42:13.030264 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:13.030264 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:42:13.030264 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:42:13.030264 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:13.030264 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:42:13.030264 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:13.030264 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:42:13.031096 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.040 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:13.031096 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:42:13.031096 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:42:13.031096 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:42:13.031096 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:42:13.031096 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:42:13.031096 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:42:13.031096 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:13.031096 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:42:13.031096 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:42:13.031096 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:42:13.031096 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:13.031763 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:42:13.031763 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:42:13.031763 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:42:13.031763 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:42:13.031763 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:42:13.031763 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:42:13.031763 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:42:13.031763 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:42:13.031763 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:13.031763 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:42:13.031763 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:42:13.031763 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:13.031763 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:42:13.032381 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:13.032381 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:42:13.032381 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:13.032381 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:42:13.032381 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:42:13.032381 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:42:13.032381 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:42:13.032381 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:42:13.032381 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:42:13.032381 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:42:13.032381 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:42:13.032381 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:42:13.032381 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:42:13.032381 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:42:13.032381 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:42:13.033160 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:42:13.033160 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:42:13.033160 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:42:13.033160 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:42:13.033160 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:42:13.033160 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:13.033160 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:42:13.033160 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:42:13.033160 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:42:13.033160 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:13.033160 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:42:13.033160 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:42:13.033160 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:42:13.033160 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:42:13.033160 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:42:13.033160 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:42:13.033160 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:42:13.033160 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:42:13.033160 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:13.034041 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:42:13.034041 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:42:13.034041 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:13.034041 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:42:13.034041 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:13.034041 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:42:13.453179 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:42:13.454116 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:42:13.454294 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:42:13.455459 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:42:13.455827 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:13.456847 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:42:13.456847 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:13.456847 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:42:13.456847 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:42:13.456847 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:13.456847 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:42:13.456847 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:13.456847 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:42:13.458612 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:13.458612 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:42:13.458612 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:42:13.458612 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:42:13.458612 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:42:13.458612 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:42:13.458612 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:42:13.458612 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:13.458612 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:42:13.458612 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:42:13.458612 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:42:13.458612 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:13.459189 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:42:13.459189 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:42:13.459189 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:42:13.459189 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:42:13.459189 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:42:13.459189 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:42:13.459189 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:42:13.459189 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:42:13.459189 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:13.459189 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:42:13.459189 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:42:13.459189 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:13.459189 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:42:13.459732 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:13.459732 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:42:13.460247 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:13.460247 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:42:13.460247 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:42:13.460247 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:42:13.460247 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:42:13.460247 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:42:13.460247 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:42:13.460247 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:42:13.460247 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:42:13.460247 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:42:13.460247 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:42:13.460247 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:42:13.460247 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:42:13.460247 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:42:13.460247 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:42:13.460247 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:42:13.460958 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:42:13.460958 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:42:13.460958 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:13.460958 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:42:13.460958 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:42:13.460958 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:42:13.460958 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:13.460958 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:42:13.460958 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:42:13.460958 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:42:13.460958 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:42:13.460958 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:42:13.460958 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:42:13.460958 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:42:13.460958 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:42:13.460958 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:13.460958 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:42:13.460958 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:42:13.460958 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:13.461840 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:42:13.461840 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:13.461840 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:42:13.464309 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:42:13.465929 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:42:13.465929 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:42:13.465929 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:42:13.465929 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:42:13.465929 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:13.466604 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:42:13.466604 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:13.466604 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:42:13.466604 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:42:13.466604 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:13.466604 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:42:13.466604 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:13.466604 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:42:13.467407 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:13.467407 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:42:13.467407 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:42:13.467407 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:42:13.467407 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:13.467407 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:42:13.467407 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:42:13.467407 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:42:13.467407 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:13.467407 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:42:13.467407 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:42:13.467407 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:42:13.467407 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:42:13.467407 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:42:13.467407 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:42:13.468071 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:42:13.468071 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:42:13.468071 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:13.468071 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:42:13.468071 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:42:13.468071 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:13.468071 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:42:13.468071 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:13.468071 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:42:13.468356 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:13.468356 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:42:13.468356 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:42:13.468356 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:42:13.468356 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:42:13.468356 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:42:13.468356 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:42:13.468356 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:42:13.468356 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:42:13.468356 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:42:13.468356 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:13.468356 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:42:13.468744 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:42:13.468744 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:42:13.468744 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:13.468744 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:42:13.468744 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:42:13.468744 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:42:13.468744 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:42:13.468744 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:42:13.468744 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:42:13.468744 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:42:13.468744 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:42:13.468744 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:13.468744 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:42:13.468744 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:42:13.469156 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:13.469156 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:42:13.469156 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:13.469156 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:42:13.469343 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:13.469343 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:42:13.469343 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:42:13.469343 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:42:13.469343 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:42:13.469343 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:42:13.469343 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:42:13.469343 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:42:13.469343 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:42:13.469343 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:42:13.469343 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:42:13.469343 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:42:13.469343 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:42:13.469343 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:42:13.469343 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:42:13.469343 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:42:13.469878 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:42:13.469878 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:42:13.469878 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:42:13.469878 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:42:13.469878 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:42:13.469878 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:13.469878 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:42:13.469878 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:42:13.469878 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:42:13.469878 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:13.469878 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:42:13.469878 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:42:13.469878 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:42:13.469878 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:42:13.469878 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:42:13.469878 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:42:13.469878 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:42:13.469878 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:42:13.470394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:13.470394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:42:13.470394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:42:13.470394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:13.470394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:42:13.470394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:13.470394 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:42:13.470394 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:42:13.470394 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:42:13.470394 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:42:13.470666 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:42:13.470666 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:42:13.470860 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:13.471383 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:42:13.471383 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:13.471383 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:42:13.471383 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:42:13.471383 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:13.471383 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:42:13.471383 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:13.471383 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:42:13.472173 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:13.472173 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:42:13.472173 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:42:13.472173 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:42:13.472173 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:13.472173 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:42:13.472173 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:42:13.472173 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:42:13.472173 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:13.472173 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:42:13.472173 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:42:13.472173 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:42:13.472173 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:42:13.472173 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:42:13.472173 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:42:13.472639 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:42:13.472639 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:42:13.472639 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:13.472639 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:42:13.472639 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:42:13.472639 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:13.472639 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:42:13.472639 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:13.472639 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:42:13.473146 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:13.473146 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:42:13.473146 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:42:13.473146 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:42:13.473146 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:42:13.473146 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:42:13.473146 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:42:13.473146 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:42:13.473146 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:42:13.473146 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:13.473146 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:42:13.473539 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:42:13.473539 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:42:13.473539 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:13.473539 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:42:13.473539 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:42:13.473539 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:42:13.473539 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:42:13.473539 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:42:13.473539 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:42:13.473539 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:42:13.473539 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:42:13.473539 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:13.473539 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:42:13.473539 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:42:13.473929 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:13.473929 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:42:13.473929 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:13.473929 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:42:13.474238 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:13.474238 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:42:13.474238 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:42:13.474238 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:42:13.474238 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:42:13.474238 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:42:13.474238 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:42:13.474238 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:42:13.474238 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:42:13.474238 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:42:13.474238 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:42:13.474238 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:42:13.474238 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:42:13.474238 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:42:13.474238 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:42:13.474238 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:42:13.474238 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:42:13.474756 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:42:13.474756 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:42:13.474756 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:42:13.474756 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:13.474756 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:42:13.474756 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:42:13.474756 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:42:13.474756 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:13.474756 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:42:13.474756 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:42:13.474756 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:42:13.474756 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:42:13.474756 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:42:13.474756 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:42:13.474756 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:42:13.474756 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:42:13.474756 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:13.474756 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:42:13.474756 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:42:13.475294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:13.475294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:42:13.475294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:13.475294 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:42:13.475294 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:42:13.475294 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:42:13.475294 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:42:13.475294 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:42:13.475848 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:42:13.476131 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:42:13.476334 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:13.476888 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:42:13.476888 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:13.476888 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:42:13.476888 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:42:13.476888 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:13.476888 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:42:13.476888 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:13.476888 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:42:13.477802 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:13.477802 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:42:13.477802 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:42:13.477802 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:42:13.477802 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:42:13.477802 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:42:13.477802 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:42:13.477802 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:13.477802 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:42:13.477802 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:42:13.477802 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:42:13.477802 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:13.478057 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:42:13.478057 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:42:13.478057 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:42:13.478057 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:42:13.478057 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:42:13.478057 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:42:13.478057 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:42:13.478057 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:42:13.478057 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:13.478057 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:42:13.478057 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:42:13.478057 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:13.478057 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:42:13.478306 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:13.478306 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:42:13.478819 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:13.478819 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:42:13.478819 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:42:13.478819 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:42:13.478819 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:42:13.478819 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:42:13.478819 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:42:13.478819 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:42:13.478819 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:42:13.478819 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:42:13.478819 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:42:13.478819 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:42:13.478819 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:42:13.478819 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:42:13.478819 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:42:13.478819 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:42:13.479127 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:42:13.479127 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:42:13.479127 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:13.479127 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:42:13.479127 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:42:13.479127 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:42:13.479127 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:13.479127 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:42:13.479127 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:42:13.479127 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:42:13.479127 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:42:13.479127 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:42:13.479127 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:42:13.479127 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:42:13.479127 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:42:13.479127 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:13.479127 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:42:13.479127 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:42:13.479127 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:13.479533 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:42:13.479533 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:13.479533 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:42:17.990447 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:42:17.991269 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:42:17.991889 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:42:17.992860 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:42:18.025858 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:42:18.026297 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:18.027439 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:42:18.027439 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:18.027439 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:42:18.027439 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:42:18.027439 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:18.027439 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:42:18.027439 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:18.027439 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:42:18.029282 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.036 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:18.029282 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:42:18.029282 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:42:18.029282 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:42:18.029282 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:42:18.029282 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:42:18.029282 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:42:18.029282 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:18.029282 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:42:18.029282 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:42:18.029282 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:42:18.029282 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:18.029925 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:42:18.029925 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:42:18.029925 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:42:18.029925 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:42:18.029925 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:42:18.029925 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:42:18.029925 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:42:18.029925 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:42:18.029925 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:18.029925 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:42:18.029925 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:42:18.029925 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:18.029925 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:42:18.030754 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:18.030754 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:42:18.031622 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:18.031622 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:42:18.031622 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:42:18.031622 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:42:18.031622 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:42:18.031622 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:42:18.031622 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:42:18.031622 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:42:18.031622 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:42:18.031622 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:42:18.031622 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:42:18.031622 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:42:18.031622 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:42:18.031622 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:42:18.031622 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:42:18.031622 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:42:18.033209 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:42:18.033209 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:42:18.033209 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:18.033209 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:42:18.033209 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:42:18.033209 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:42:18.033209 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:18.033209 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:42:18.033209 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:42:18.033209 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:42:18.033209 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:42:18.033209 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:42:18.033209 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:42:18.033209 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:42:18.033209 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:42:18.033209 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:18.033209 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:42:18.033209 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:42:18.033209 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:18.034887 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:42:18.034887 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:18.034887 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:42:18.454074 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:42:18.454695 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:42:18.454925 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:42:18.456087 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:42:18.456475 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:18.457400 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:42:18.457400 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:18.457400 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:42:18.457400 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:42:18.457400 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:18.457400 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:42:18.457400 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:18.457400 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:42:18.458963 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:18.458963 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:42:18.458963 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:42:18.458963 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:42:18.458963 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:42:18.458963 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:42:18.458963 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:42:18.458963 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:18.458963 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:42:18.458963 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:42:18.458963 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:42:18.458963 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:18.459547 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:42:18.459547 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:42:18.459547 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:42:18.459547 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:42:18.459547 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:42:18.459547 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:42:18.459547 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:42:18.459547 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:42:18.459547 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:18.459547 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:42:18.459547 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:42:18.459547 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:18.459547 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:42:18.460113 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:18.460113 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:42:18.460741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:18.460741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:42:18.460741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:42:18.460741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:42:18.460741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:42:18.460741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:42:18.460741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:42:18.460741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:42:18.460741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:42:18.460741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:42:18.460741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:42:18.460741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:42:18.460741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:42:18.460741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:42:18.460741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:42:18.460741 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:42:18.462268 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:42:18.462268 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:42:18.462268 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:18.462268 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:42:18.462268 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:42:18.462268 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:42:18.462268 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:18.462268 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:42:18.462268 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:42:18.462268 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:42:18.462268 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:42:18.462268 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:42:18.462268 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:42:18.462268 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:42:18.462268 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:42:18.462268 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:18.462268 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:42:18.462268 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:42:18.462268 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:18.463808 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:42:18.463808 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:18.463808 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:42:18.464974 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:42:18.465409 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:42:18.465618 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:42:18.465805 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:42:18.466175 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:42:18.466414 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:18.467045 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:42:18.467045 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:18.467045 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:42:18.467045 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:42:18.467045 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:18.467045 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:42:18.467045 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:18.467045 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:42:18.467901 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:18.467901 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:42:18.467901 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:42:18.467901 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:42:18.467901 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:18.467901 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:42:18.467901 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:42:18.467901 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:42:18.467901 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:18.467901 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:42:18.467901 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:42:18.467901 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:42:18.467901 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:42:18.467901 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:42:18.467901 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:42:18.468280 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:42:18.468280 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:42:18.468280 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:18.468280 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:42:18.468280 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:42:18.468280 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:18.468280 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:42:18.468280 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:18.468280 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:42:18.468826 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:18.468826 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:42:18.468826 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:42:18.468826 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:42:18.468826 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:42:18.468826 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:42:18.468826 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:42:18.468826 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:42:18.468826 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:42:18.468826 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:42:18.468826 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:18.468826 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:42:18.469487 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:42:18.469487 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:42:18.469487 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:18.469487 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:42:18.469487 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:42:18.469487 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:42:18.469487 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:42:18.469487 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:42:18.469487 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:42:18.469487 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:42:18.469487 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:42:18.469487 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:18.469487 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:42:18.469487 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:42:18.470145 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:18.470145 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:42:18.470145 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:18.470145 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:42:18.470145 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:18.470145 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:42:18.470145 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:42:18.470145 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:42:18.470145 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:42:18.470145 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:42:18.470145 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:42:18.470145 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:42:18.470145 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:42:18.470145 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:42:18.470499 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:42:18.470499 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:42:18.470499 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:42:18.470499 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:42:18.470499 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:42:18.470499 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:42:18.470499 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:42:18.470499 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:42:18.470499 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:42:18.470499 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:42:18.470499 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:42:18.470499 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:18.470499 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:42:18.470499 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:42:18.470499 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:42:18.470499 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:18.470499 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:42:18.470499 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:42:18.470499 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:42:18.470966 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:42:18.470966 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:42:18.470966 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:42:18.470966 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:42:18.470966 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:42:18.470966 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:18.470966 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:42:18.470966 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:42:18.470966 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:18.470966 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:42:18.470966 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:18.470966 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:42:18.470966 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:42:18.470966 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:42:18.471311 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:42:18.471311 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:42:18.471311 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:42:18.471311 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:18.471807 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:42:18.471807 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:18.471807 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:42:18.471807 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:42:18.471807 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:18.471807 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:42:18.471807 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:18.471807 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:42:18.472546 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:18.472546 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:42:18.472546 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:42:18.472546 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:42:18.472546 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:18.472546 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:42:18.472546 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:42:18.472546 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:42:18.472546 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:18.472546 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:42:18.472546 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:42:18.472546 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:42:18.472546 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:42:18.472546 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:42:18.472546 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:42:18.472927 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:42:18.472927 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:42:18.472927 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:18.472927 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:42:18.472927 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:42:18.472927 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:18.472927 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:42:18.472927 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:18.472927 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:42:18.473452 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:18.473452 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:42:18.473452 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:42:18.473452 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:42:18.473452 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:42:18.473452 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:42:18.473452 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:42:18.473452 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:42:18.473452 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:42:18.473452 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:18.473452 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:42:18.474054 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:42:18.474054 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:42:18.474054 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:18.474054 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:42:18.474054 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:42:18.474054 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:42:18.474054 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:42:18.474054 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:42:18.474054 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:42:18.474054 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:42:18.474054 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:42:18.474054 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:18.474054 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:42:18.474054 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:42:18.474917 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:18.474917 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:42:18.474917 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:18.474917 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:42:18.474917 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:18.474917 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:42:18.474917 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:42:18.474917 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:42:18.474917 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:42:18.474917 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:42:18.474917 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:42:18.474917 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:42:18.474917 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:42:18.474917 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:42:18.475251 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:42:18.475251 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:42:18.475251 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:42:18.475251 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:42:18.475251 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:42:18.475251 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:42:18.475251 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:42:18.475251 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:42:18.475251 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:42:18.475251 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:42:18.475251 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:18.475251 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:42:18.475251 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:42:18.475251 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:42:18.475251 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:18.475251 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:42:18.475251 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:42:18.475251 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:42:18.475251 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:42:18.475712 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:42:18.475712 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:42:18.475712 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:42:18.475712 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:42:18.475712 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:18.475712 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:42:18.475712 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:42:18.475712 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:18.475712 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:42:18.475712 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:18.475712 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:42:18.475712 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:42:18.475712 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:42:18.476035 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:42:18.476035 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:42:18.476035 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:42:18.476583 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:42:18.476583 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:18.476813 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:42:18.476813 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:18.476813 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:42:18.476813 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:42:18.476813 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:18.476813 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:42:18.476813 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:18.476813 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:42:18.477401 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:18.477401 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:42:18.477401 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:42:18.477401 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:42:18.477401 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:42:18.477401 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:42:18.477401 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:42:18.477401 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:18.477401 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:42:18.477401 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:42:18.477401 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:42:18.477401 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:18.477924 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:42:18.477924 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:42:18.477924 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:42:18.477924 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:42:18.477924 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:42:18.477924 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:42:18.477924 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:42:18.477924 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:42:18.477924 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:18.477924 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:42:18.477924 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:42:18.477924 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:18.477924 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:42:18.478432 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:18.478432 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:42:18.478432 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:18.478432 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:42:18.478432 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:42:18.478432 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:42:18.478432 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:42:18.478432 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:42:18.478432 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:42:18.478432 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:42:18.478432 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:42:18.478432 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:42:18.478432 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:42:18.478432 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:42:18.478432 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:42:18.478874 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:42:18.478874 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:42:18.478874 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:42:18.478874 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:42:18.478874 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:42:18.478874 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:18.478874 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:42:18.478874 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:42:18.478874 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:42:18.478874 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:18.478874 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:42:18.478874 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:42:18.478874 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:42:18.478874 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:42:18.478874 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:42:18.478874 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:42:18.478874 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:42:18.478874 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:42:18.478874 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:18.479210 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:42:18.479210 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:42:18.479210 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:18.479210 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:42:18.479210 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:18.479210 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:42:22.991018 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:42:22.991962 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:42:22.991962 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:42:22.992830 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:42:23.022403 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:42:23.022624 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:23.023020 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:42:23.023020 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:23.023020 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:42:23.023020 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:42:23.023020 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:23.023020 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:42:23.023020 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:23.023020 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:42:23.023745 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.031 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:23.023745 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:42:23.023745 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:42:23.023745 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:42:23.023745 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:42:23.023745 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:42:23.023745 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:42:23.023745 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:23.023745 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:42:23.023745 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:42:23.023745 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:42:23.023745 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:23.024843 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:42:23.024843 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:42:23.024843 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:42:23.024843 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:42:23.024843 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:42:23.024843 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:42:23.024843 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:42:23.024843 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:42:23.024843 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:23.024843 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:42:23.024843 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:42:23.024843 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:23.024843 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:42:23.025162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:23.025162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:42:23.025162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:23.025162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:42:23.025162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:42:23.025162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:42:23.025162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:42:23.025162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:42:23.025162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:42:23.025162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:42:23.025162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:42:23.025162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:42:23.025162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:42:23.025162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:42:23.025162 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:42:23.025519 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:42:23.025519 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:42:23.025519 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:42:23.025519 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:42:23.025519 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:42:23.025519 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:23.025519 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:42:23.025519 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:42:23.025519 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:42:23.025519 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:23.025519 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:42:23.025519 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:42:23.025519 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:42:23.025519 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:42:23.025519 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:42:23.025519 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:42:23.025519 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:42:23.025519 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:42:23.025519 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:23.025934 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:42:23.025934 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:42:23.025934 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:23.025934 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:42:23.025934 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:23.025934 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:42:23.454841 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:42:23.455483 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:42:23.455675 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:42:23.456577 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:42:23.456915 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:23.457768 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:42:23.457768 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:23.457768 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:42:23.457768 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:42:23.457768 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:23.457768 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:42:23.457768 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:23.457768 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:42:23.459419 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:23.459419 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:42:23.459419 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:42:23.459419 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:42:23.459419 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:42:23.459419 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:42:23.459419 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:42:23.459419 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:23.459419 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:42:23.459419 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:42:23.459419 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:42:23.459419 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:23.459763 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:42:23.459763 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:42:23.459763 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:42:23.459763 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:42:23.459763 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:42:23.459763 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:42:23.459763 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:42:23.459763 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:42:23.459763 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:23.459763 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:42:23.459763 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:42:23.459763 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:23.459763 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:42:23.460078 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:23.460078 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:42:23.461806 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:23.461806 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:42:23.461806 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:42:23.461806 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:42:23.461806 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:42:23.461806 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:42:23.461806 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:42:23.461806 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:42:23.461806 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:42:23.461806 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:42:23.461806 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:42:23.461806 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:42:23.461806 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:42:23.461806 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:42:23.461806 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:42:23.461806 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:42:23.462371 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:42:23.462371 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:42:23.462371 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:23.462371 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:42:23.462371 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:42:23.462371 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:42:23.462371 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:23.462371 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:42:23.462371 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:42:23.462371 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:42:23.462371 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:42:23.462371 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:42:23.462371 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:42:23.462371 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:42:23.462371 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:42:23.462371 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:23.462371 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:42:23.462371 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:42:23.462371 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:23.462980 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:42:23.462980 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:23.462980 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:42:23.465473 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:42:23.465768 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:42:23.465903 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:42:23.466053 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:42:23.466591 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:42:23.466917 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:23.467685 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:42:23.467685 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:23.467685 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:42:23.467685 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:42:23.467685 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:23.467685 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:42:23.467685 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:23.467685 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:42:23.468455 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:23.468455 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:42:23.468455 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:42:23.468455 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:42:23.468455 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:23.468455 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:42:23.468455 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:42:23.468455 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:42:23.468455 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:23.468455 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:42:23.468455 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:42:23.468455 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:42:23.468455 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:42:23.468455 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:42:23.468455 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:42:23.468905 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:42:23.468905 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:42:23.468905 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:23.468905 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:42:23.468905 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:42:23.468905 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:23.468905 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:42:23.468905 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:23.468905 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:42:23.469470 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:23.469470 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:42:23.469470 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:42:23.469470 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:42:23.469470 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:42:23.469470 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:42:23.469470 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:42:23.469470 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:42:23.469470 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:42:23.469470 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:42:23.469470 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:23.469470 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:42:23.470186 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:42:23.470186 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:42:23.470186 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:23.470186 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:42:23.470186 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:42:23.470186 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:42:23.470186 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:42:23.470186 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:42:23.470186 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:42:23.470186 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:42:23.470186 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:42:23.470186 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:23.470186 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:42:23.470186 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:42:23.470993 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:23.470993 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:42:23.470993 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:23.470993 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:42:23.470993 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:23.470993 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:42:23.470993 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:42:23.470993 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:42:23.470993 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:42:23.470993 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:42:23.470993 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:42:23.470993 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:42:23.470993 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:42:23.470993 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:42:23.471346 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:42:23.471346 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:42:23.471346 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:42:23.471346 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:42:23.471346 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:42:23.471346 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:42:23.471346 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:42:23.471346 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:42:23.471346 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:42:23.471346 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:42:23.471346 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:42:23.471346 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:23.471346 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:42:23.471346 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:42:23.471346 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:42:23.471346 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:23.471346 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:42:23.471346 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:42:23.471346 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:42:23.471833 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:42:23.471833 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:42:23.471833 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:42:23.471833 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:42:23.471833 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:42:23.471833 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:23.471833 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:42:23.471833 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:42:23.471833 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:23.471833 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:42:23.471833 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:23.471833 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:42:23.471833 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:42:23.471833 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:42:23.472185 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:42:23.472185 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:42:23.472185 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:42:23.472690 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:23.473192 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:42:23.473192 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:23.473192 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:42:23.473192 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:42:23.473192 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:23.473192 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:42:23.473192 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:23.473192 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:42:23.474055 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:23.474055 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:42:23.474055 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:42:23.474055 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:42:23.474055 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:23.474055 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:42:23.474055 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:42:23.474055 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:42:23.474055 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:23.474055 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:42:23.474055 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:42:23.474055 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:42:23.474055 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:42:23.474055 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:42:23.474055 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:42:23.474933 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:42:23.474933 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:42:23.474933 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:23.474933 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:42:23.474933 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:42:23.474933 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:23.474933 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:42:23.474933 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:23.474933 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:42:23.475458 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:23.475458 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:42:23.475458 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:42:23.475458 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:42:23.475458 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:42:23.475458 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:42:23.475458 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:42:23.475458 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:42:23.475458 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:42:23.475458 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:23.475458 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:42:23.475926 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:42:23.475926 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:42:23.475926 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:23.475926 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:42:23.475926 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:42:23.475926 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:42:23.475926 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:42:23.475926 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:42:23.475926 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:42:23.475926 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:42:23.475926 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:42:23.475926 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:23.475926 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:42:23.475926 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:42:23.476256 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:23.476256 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:42:23.476256 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:23.476256 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:42:23.476731 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:23.476731 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:42:23.476731 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:42:23.476731 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:42:23.476731 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:42:23.476731 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:42:23.476731 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:42:23.476731 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:42:23.476731 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:42:23.476731 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:42:23.476731 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:42:23.476731 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:42:23.476731 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:42:23.476731 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:42:23.476731 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:42:23.476731 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:42:23.476731 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:42:23.477308 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:42:23.477308 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:42:23.477308 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:42:23.477308 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:23.477308 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:42:23.477308 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:42:23.477308 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:42:23.477308 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:23.477308 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:42:23.477308 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:42:23.477308 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:42:23.477308 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:42:23.477308 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:42:23.477308 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:42:23.477308 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:42:23.477308 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:42:23.477308 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:23.477308 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:42:23.477308 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:42:23.477852 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:23.477852 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:42:23.477852 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:23.477852 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:42:23.477852 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:42:23.477852 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:42:23.477852 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:42:23.477852 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:42:23.478482 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:42:23.478482 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:42:23.478698 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:23.479377 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:42:23.479377 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:23.479377 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:42:23.479377 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:42:23.479377 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:23.479377 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:42:23.479377 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:23.479377 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:42:23.480287 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:23.480287 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:42:23.480287 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:42:23.480287 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:42:23.480287 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:42:23.480287 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:42:23.480287 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:42:23.480287 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:23.480287 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:42:23.480287 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:42:23.480287 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:42:23.480287 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:23.481033 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:42:23.481033 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:42:23.481033 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:42:23.481033 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:42:23.481033 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:42:23.481033 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:42:23.481033 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:42:23.481033 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:42:23.481033 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:23.481033 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:42:23.481033 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:42:23.481033 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:23.481033 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:42:23.481815 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:23.481815 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:42:23.481815 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:23.481815 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:42:23.481815 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:42:23.481815 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:42:23.481815 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:42:23.481815 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:42:23.481815 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:42:23.481815 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:42:23.481815 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:42:23.481815 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:42:23.481815 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:42:23.481815 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:42:23.481815 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:42:23.482517 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:42:23.482517 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:42:23.482517 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:42:23.482517 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:42:23.482517 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:42:23.482517 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:23.482517 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:42:23.482517 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:42:23.482517 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:42:23.482517 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:23.482517 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:42:23.482517 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:42:23.482517 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:42:23.482517 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:42:23.482517 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:42:23.482517 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:42:23.482517 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:42:23.482517 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:42:23.482517 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:23.483082 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:42:23.483082 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:42:23.483082 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:23.483082 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:42:23.483082 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:23.483082 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:42:27.991997 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:42:27.992891 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:42:27.992891 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:42:27.994242 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:42:28.031119 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:42:28.031119 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:28.032194 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:42:28.032194 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:28.032194 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:42:28.032194 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:42:28.032194 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:28.032194 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:42:28.032194 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:28.032194 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:42:28.033415 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_fdb_aging_settings failed after 0.039 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:28.033415 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:42:28.033415 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:42:28.033415 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:42:28.033415 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:42:28.033415 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:42:28.033415 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:42:28.033415 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:28.033415 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:42:28.033415 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:42:28.033415 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:42:28.033415 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:28.033982 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:42:28.033982 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:42:28.033982 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:42:28.033982 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:42:28.033982 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:42:28.033982 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:42:28.033982 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:42:28.033982 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:42:28.033982 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:28.033982 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:42:28.033982 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:42:28.033982 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:28.033982 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:42:28.034572 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:28.034572 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:42:28.034895 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:28.034895 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:42:28.034895 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:42:28.034895 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:42:28.034895 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:42:28.034895 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:42:28.034895 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:42:28.034895 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:42:28.034895 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:42:28.034895 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:42:28.034895 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:42:28.034895 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:42:28.034895 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:42:28.034895 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:42:28.034895 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:42:28.034895 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:42:28.035219 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 18 18:42:28.035219 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:42:28.035219 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:28.035219 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:42:28.035219 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:42:28.035219 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:42:28.035219 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:28.035219 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:42:28.035219 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:42:28.035219 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:42:28.035219 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:42:28.035219 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:42:28.035219 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:42:28.035219 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:42:28.035219 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:42:28.035219 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:28.035219 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:42:28.035219 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:42:28.035219 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:28.035586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:42:28.035586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:28.035586 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:42:28.455623 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:42:28.456382 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:42:28.456610 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:42:28.457703 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3eb9790c-f309-4e4a-a7b2-f3e31a4aa573, routes=[], if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:42:28.458102 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:28.459158 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:42:28.459158 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:28.459158 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:42:28.459158 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:42:28.459158 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:28.459158 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:42:28.459158 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:28.459158 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:42:28.460840 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:28.460840 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:42:28.460840 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:42:28.460840 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:42:28.460840 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:42:28.460840 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:42:28.460840 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:42:28.460840 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:28.460840 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:42:28.460840 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:42:28.460840 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:42:28.460840 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:28.461448 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:42:28.461448 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:42:28.461448 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:42:28.461448 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:42:28.461448 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:42:28.461448 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:42:28.461448 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:42:28.461448 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:42:28.461448 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:28.461448 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:42:28.461448 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:42:28.461448 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:28.461448 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:42:28.462076 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:28.462076 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:42:28.462787 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:28.462787 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:42:28.462787 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:42:28.462787 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:42:28.462787 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:42:28.462787 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:42:28.462787 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:42:28.462787 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:42:28.462787 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:42:28.462787 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:42:28.462787 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:42:28.462787 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:42:28.462787 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:42:28.462787 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:42:28.462787 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:42:28.462787 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:42:28.463934 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 18 18:42:28.463934 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:42:28.463934 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:28.463934 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:42:28.463934 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:42:28.463934 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:42:28.463934 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:28.463934 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:42:28.463934 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:42:28.463934 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:42:28.463934 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:42:28.463934 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:42:28.463934 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:42:28.463934 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:42:28.463934 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:42:28.463934 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:28.463934 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:42:28.463934 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:42:28.463934 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:28.465183 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:42:28.465183 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:28.465183 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:42:28.465892 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:42:28.466271 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:42:28.466541 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:42:28.466747 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting fip-distributed flag in NB_Global to False {{(pid=106123) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 18 18:42:28.467224 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:42:28.467564 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:28.468253 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:42:28.468253 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:28.468253 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:42:28.468253 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:42:28.468253 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:28.468253 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:42:28.468253 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:28.468253 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:42:28.469071 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:28.469071 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:42:28.469071 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:42:28.469071 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:42:28.469071 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:28.469071 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:42:28.469071 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:42:28.469071 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:42:28.469071 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:28.469071 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:42:28.469071 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:42:28.469071 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:42:28.469071 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:42:28.469071 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:42:28.469071 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:42:28.469553 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:42:28.469553 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:42:28.469553 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:28.469553 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:42:28.469553 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:42:28.469553 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:28.469553 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:42:28.469553 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:28.469553 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:42:28.470044 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:28.470044 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:42:28.470044 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:42:28.470044 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:42:28.470044 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:42:28.470044 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:42:28.470044 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:42:28.470044 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 18 18:42:28.470044 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:42:28.470044 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:42:28.470044 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:28.470044 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:42:28.470956 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:42:28.470956 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:42:28.470956 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:28.470956 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:42:28.470956 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:42:28.470956 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:42:28.470956 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:42:28.470956 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:42:28.470956 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:42:28.470956 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:42:28.470956 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:42:28.470956 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:28.470956 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:42:28.470956 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:42:28.471661 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:28.471661 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:42:28.471661 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:28.471661 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:42:28.471661 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:28.471661 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:42:28.471661 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:42:28.471661 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:42:28.471661 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:42:28.471661 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:42:28.471661 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:42:28.471661 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:42:28.471661 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:42:28.471661 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:42:28.472024 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:42:28.472024 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:42:28.472024 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:42:28.472024 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:42:28.472024 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:42:28.472024 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:42:28.472024 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 18 18:42:28.472024 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 18 18:42:28.472024 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^ May 18 18:42:28.472024 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:42:28.472024 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:42:28.472024 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:28.472024 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:42:28.472024 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:42:28.472024 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:42:28.472024 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:28.472024 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:42:28.472024 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:42:28.472024 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:42:28.472539 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:42:28.472539 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:42:28.472539 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:42:28.472539 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:42:28.472539 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:42:28.472539 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:28.472539 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:42:28.472539 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:42:28.472539 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:28.472539 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:42:28.472539 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:28.472539 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:42:28.472539 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:42:28.472539 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_ha_failover {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:42:28.472923 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:42:28.472923 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:42:28.473596 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:42:28.474049 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:28.474986 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:42:28.474986 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:28.474986 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:42:28.474986 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:42:28.474986 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:28.474986 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:42:28.474986 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:28.474986 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:42:28.475757 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:28.475757 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 18 18:42:28.475757 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:42:28.475757 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:42:28.475757 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:28.475757 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:42:28.475757 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:42:28.475757 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 18 18:42:28.475757 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:28.475757 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 18 18:42:28.475757 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:42:28.475757 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 18 18:42:28.475757 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:42:28.475757 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 18 18:42:28.475757 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 18 18:42:28.476721 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:42:28.476721 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 18 18:42:28.476721 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:28.476721 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 18 18:42:28.476721 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 18 18:42:28.476721 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:28.476721 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 18 18:42:28.476721 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:28.476721 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.command May 18 18:42:28.476721 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:28.476721 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:42:28.476721 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:42:28.476721 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:42:28.476721 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:42:28.478167 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:42:28.478167 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 18 18:42:28.478167 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:42:28.478167 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:42:28.478167 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:28.478167 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:42:28.478167 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:42:28.478167 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:42:28.478167 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:28.478167 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:42:28.478167 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:42:28.478167 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:42:28.478167 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:42:28.478837 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:42:28.478837 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:42:28.478837 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:42:28.478837 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:42:28.478837 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:28.478837 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:42:28.478837 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:42:28.478837 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:28.478837 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:42:28.478837 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:28.478837 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:42:28.478837 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:28.479268 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:42:28.479268 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:42:28.479268 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:42:28.479268 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:42:28.479268 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:42:28.479268 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:42:28.479268 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:42:28.479268 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:42:28.479268 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:42:28.479268 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:42:28.479268 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:42:28.479268 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:42:28.479268 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:42:28.479268 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:42:28.479268 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:42:28.479268 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 18 18:42:28.479268 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 18 18:42:28.479268 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 18 18:42:28.479268 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 18 18:42:28.480264 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:28.480264 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:42:28.480264 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:42:28.480264 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:42:28.480264 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:28.480264 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:42:28.480264 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:42:28.480264 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:42:28.480264 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:42:28.480264 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:42:28.480264 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:42:28.480264 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:42:28.480264 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:42:28.480264 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:28.480264 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:42:28.480264 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:42:28.480264 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:28.480264 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:42:28.480264 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:28.480264 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics May 18 18:42:28.480809 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=106123) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 18 18:42:28.480809 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 18 18:42:28.480809 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=106123) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 18 18:42:28.480809 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=106123) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 18 18:42:28.480809 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:42:28.480809 np0000174525 neutron-ovn-maintenance-worker[106123]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=106123) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 18 18:42:28.480987 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:28.480987 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Traceback (most recent call last): May 18 18:42:28.480987 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:28.480987 np0000174525 neutron-ovn-maintenance-worker[106123]: txn.results.put(txn.do_commit()) May 18 18:42:28.480987 np0000174525 neutron-ovn-maintenance-worker[106123]: ^^^^^^^^^^^^^^^ May 18 18:42:28.480987 np0000174525 neutron-ovn-maintenance-worker[106123]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:28.480987 np0000174525 neutron-ovn-maintenance-worker[106123]: raise RuntimeError(msg) May 18 18:42:28.480987 np0000174525 neutron-ovn-maintenance-worker[106123]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:28.480987 np0000174525 neutron-ovn-maintenance-worker[106123]: May 18 18:42:28.481581 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:28.481581 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 18 18:42:28.481581 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:42:28.481581 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 18 18:42:28.481581 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 18 18:42:28.481581 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:42:28.481581 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 18 18:42:28.481581 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:28.481581 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:42:28.481581 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:42:28.481581 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 18 18:42:28.481581 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:28.481947 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 18 18:42:28.481947 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:42:28.481947 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 18 18:42:28.481947 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:42:28.481947 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 18 18:42:28.481947 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 18 18:42:28.481947 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:42:28.481947 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 18 18:42:28.481947 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:28.481947 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 18 18:42:28.481947 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 18 18:42:28.481947 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:28.481947 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 18 18:42:28.482515 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:28.482515 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 18 18:42:28.482515 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics [None req-eae1c660-058d-49ca-9b8c-3eb5b14b0668 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:28.482515 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics Traceback (most recent call last): May 18 18:42:28.482515 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 18 18:42:28.482515 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics work() May 18 18:42:28.482515 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 18 18:42:28.482515 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 18 18:42:28.482515 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:42:28.482515 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 18 18:42:28.482515 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(*args, **kwargs) May 18 18:42:28.482515 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:42:28.482515 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 18 18:42:28.482515 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics return f(self, *args, **kwargs) May 18 18:42:28.482515 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 18 18:42:28.482939 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 18 18:42:28.482939 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics result = f(*args, **kwargs) May 18 18:42:28.482939 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 18 18:42:28.482939 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 18 18:42:28.482939 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 18 18:42:28.482939 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:28.482939 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:42:28.482939 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 18 18:42:28.482939 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 18 18:42:28.482939 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 18 18:42:28.482939 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics next(self.gen) May 18 18:42:28.482939 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 18 18:42:28.482939 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics with self.create_transaction( May 18 18:42:28.482939 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 18 18:42:28.482939 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics self.result = self.commit() May 18 18:42:28.482939 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 18 18:42:28.482939 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 18 18:42:28.482939 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise result.ex May 18 18:42:28.482939 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 18 18:42:28.484095 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 18 18:42:28.484095 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 18 18:42:28.484095 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 18 18:42:28.484095 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics raise RuntimeError(msg) May 18 18:42:28.484095 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 18 18:42:28.484095 np0000174525 neutron-ovn-maintenance-worker[106123]: ERROR futurist.periodics